Static routes...not as expected?
-
I just noticed when my primary WAN is down, I can still access/ping stuff that is set "static route" thru the primary WAN.
Is this normal?
Static routes:
When "WAN" (the primary) is down and it's failed over to WAN2, should clients be able to reach 1.0.0.1 even though I have a static route to force that thru the WAN interface?
-
@mmiller7 What does the actual routing table show? Diagnostics > Routes
-
Simulating a failure unplugging the coax, here's what the routes table shows. Annoyingly, I was not able to reproduce the problem by pulling the cable as it happened when the conneciton failed on its own.
-
@mmiller7 Are you 100% certain the route actually has the 24.153.32.1 destination when that WAN is up?
What does executing this in Diagnostics > Command Prompt output?
grep -C10 1.0.0.1 /cf/conf/config.xmlWhile you're at it you might as well dump the full routing table:
netstat -rnfinet -
I can confirm that
Redmine issue created: https://redmine.pfsense.org/issues/11296 -
In case I forgot to say my full pfSense version before...2.4.5-RELEASE-p1 (amd64) / built on Tue Jun 02 17:51:17 EDT 2020 / FreeBSD 11.3-STABLE
@derelict Would you prefer the tests with the WAN1 up, attempting to simulate it down (pull a cable), or both? I'm assuming both may be helpful.
24.153.32.1 was the upstream gateway my ISP pushed via DHCP when I took that screenshot. I learned some time ago the hard way that it can change (as can my IP address, albeit infrequent) which is why I tried to not hard code that as the monitor IP (and I want to allow the modem self-generated 192.168.100.0/24 lease so I can see its status when it is down and use that to help complain to my ISP) so letting it pick the gateway monitor IP dynamically did not work well.
Other IPs in the screenshots:
192.168.100.1 - WAN1 Arris modem's diagnostic/status page
1.0.0.1 - WAN1 Monitor-IP - Cloudfire DNS (also WAN1 dedicated DNS)
192.168.10.1 - my WAN2 backup cellular hotspot gatewayPossibly relevant, I do have these checked in advanced settings, if that somehow affects this behavior:
From my config:
[2.4.5-RELEASE][root@pfSense.apt]/root: grep -C10 1.0.0.1 /cf/conf/config.xml <ssh> <enable>enabled</enable> </ssh> <ipv6allow></ipv6allow> <enableserial></enableserial> <afterfilterchangeshellcmd></afterfilterchangeshellcmd> <mds_disable>0</mds_disable> <gw_down_kill_states></gw_down_kill_states> <skip_rules_gw_down></skip_rules_gw_down> <dnsserver>1.1.1.1</dnsserver> <dnsserver>1.0.0.1</dnsserver> <dnsserver>8.8.8.8</dnsserver> <dnsserver>8.8.4.4</dnsserver> </system> <interfaces> <wan> <enable></enable> <if>igb0</if> <blockbogons></blockbogons> <descr><![CDATA[WAN]]></descr> <alias-address></alias-address> -- <network>192.168.100.1/32</network> <gateway>WAN_DHCP</gateway> <descr><![CDATA[Modem Diagnostics]]></descr> </route> <route> <network>192.168.10.0/24</network> <gateway>WAN2_DHCP</gateway> <descr><![CDATA[Access MiFi Settings]]></descr> </route> <route> <network>1.0.0.1/32</network> <gateway>WAN_DHCP</gateway> <descr><![CDATA[Monitoring IP and DNS for WAN]]></descr> </route> <route> <network>8.8.4.4/32</network> <gateway>WAN2_DHCP</gateway> <descr><![CDATA[Monitoring IP and DNS for WAN2]]></descr> </route> </staticroutes> <dhcpd> -- <username><![CDATA[admin@192.168.1.194 (Local Database)]]></username> </updated> <created> <time>1607182884</time> <username><![CDATA[admin@192.168.1.194 (Local Database)]]></username> </created> <disabled></disabled> </rule> <rule> <id></id> <tracker>0100000101</tracker> <type>pass</type> <interface>lan</interface> <ipprotocol>inet</ipprotocol> <tag></tag> <tagged></tagged> <max></max> <max-src-nodes></max-src-nodes> <max-src-conn></max-src-conn> <max-src-states></max-src-states> <statetimeout></statetimeout> -- <time>1607181413</time> <username><![CDATA[admin@192.168.1.194 (Local Database)]]></username> </updated> <disabled></disabled> </rule> <rule> <type>pass</type> <ipprotocol>inet6</ipprotocol> <descr><![CDATA[Default allow LAN IPv6 to any rule]]></descr> <interface>lan</interface> <tracker>0100000102</tracker> <source> <network>lan</network> </source> <destination> <any></any> </destination> </rule> <rule> <descr><![CDATA[OpenVPN wizard]]></descr> <source> -- <losshigh>75</losshigh> </gateway_item> <gateway_item> <interface>wan</interface> <gateway>dynamic</gateway> <name>WAN_DHCP</name> <weight>1</weight> <ipprotocol>inet</ipprotocol> <time_period>15000</time_period> <descr><![CDATA[Interface WAN_DHCP Gateway]]></descr> <monitor>1.0.0.1</monitor> <latencyhigh>1000</latencyhigh> <losshigh>80</losshigh> </gateway_item> <gateway_item> <interface>opt3</interface> <gateway>dynamic</gateway> <name>WAN2_DHCP6</name> <weight>1</weight> <ipprotocol>inet6</ipprotocol> <descr><![CDATA[Interface WAN2_DHCP6 Gateway]]></descr> [2.4.5-RELEASE][root@pfSense.apt]/root:I will collect the additional route information you requested.
I presume my original expectation is correct, that clients should fail to reach 1.0.0.1 if WAN1 is down, given my static route; even without fancy LAN firewall rules explicitly blocking 1.0.0.1 destination on WAN2?
-
@derelict said in Static routes...not as expected?:
@mmiller7 Are you 100% certain the route actually has the 24.153.32.1 destination when that WAN is up?
What does executing this in Diagnostics > Command Prompt output?
grep -C10 1.0.0.1 /cf/conf/config.xmlWhile you're at it you might as well dump the full routing table:
netstat -rnfinetFYI I'm having a bit of difficulty posting the information, its telling me my post was flagged for spam by some monitoring bot...hopefully the retry worked.
-
@mmiller7 You don't have to do any of that and you are probably confusing something by doing so.
Setting 1.0.0.1 as the gateway monitoring address automatically creates the host route out that interface that you are trying to create manually.
Setting 1.0.0.1 as a DNS server with a gateway set (I can't see if you have a gateway set or not) does the same thing.
There is zero need to try to make a static route like that.
-
Good news! I've reproduced my issue while capturing the requests routing info.
Routing table when WAN1 is operational:
[2.4.5-RELEASE][root@pfSense.apt]/root: netstat -rnfinet Routing tables Internet: Destination Gateway Flags Netif Expire default 24.153.32.1 UGS igb0 1.0.0.1 24.153.32.1 UGHS igb0 1.0.0.1/32 24.153.32.1 UGS igb0 8.8.4.4 192.168.10.1 UGHS igb1 8.8.4.4/32 192.168.10.1 UGS igb1 10.0.8.0/24 10.0.8.2 UGS ovpns1 10.0.8.1 link#13 UHS lo0 10.0.8.2 link#13 UH ovpns1 10.0.9.0/24 10.0.9.2 UGS ovpns2 10.0.9.1 link#14 UHS lo0 10.0.9.2 link#14 UH ovpns2 24.153.32.0/24 link#1 U igb0 24.153.32.24 link#1 UHS lo0 127.0.0.1 link#6 UH lo0 192.168.1.0/24 link#10 U lagg1 192.168.1.1 link#10 UHS lo0 192.168.2.0/24 link#11 U lagg1.2 192.168.2.1 link#11 UHS lo0 192.168.3.0/24 link#12 U lagg1.3 192.168.3.1 link#12 UHS lo0 192.168.10.0/24 192.168.10.1 UGS igb1 192.168.10.23 link#2 UHS lo0 192.168.100.1/32 24.153.32.1 UGS igb0 [2.4.5-RELEASE][root@pfSense.apt]/root:INITIALLY Routing table when WAN1 is down (pull coax from modem simulate ISP failure):
[2.4.5-RELEASE][root@pfSense.apt]/root: netstat -rnfinet Routing tables Internet: Destination Gateway Flags Netif Expire default 192.168.10.1 UGS igb1 1.0.0.1 24.153.32.1 UGHS igb0 1.0.0.1/32 24.153.32.1 UGS igb0 8.8.4.4 192.168.10.1 UGHS igb1 8.8.4.4/32 192.168.10.1 UGS igb1 10.0.8.0/24 10.0.8.2 UGS ovpns1 10.0.8.1 link#13 UHS lo0 10.0.8.2 link#13 UH ovpns1 10.0.9.0/24 10.0.9.2 UGS ovpns2 10.0.9.1 link#14 UHS lo0 10.0.9.2 link#14 UH ovpns2 24.153.32.0/24 link#1 U igb0 24.153.32.24 link#1 UHS lo0 127.0.0.1 link#6 UH lo0 192.168.1.0/24 link#10 U lagg1 192.168.1.1 link#10 UHS lo0 192.168.2.0/24 link#11 U lagg1.2 192.168.2.1 link#11 UHS lo0 192.168.3.0/24 link#12 U lagg1.3 192.168.3.1 link#12 UHS lo0 192.168.10.0/24 192.168.10.1 UGS igb1 192.168.10.23 link#2 UHS lo0 192.168.100.1/32 24.153.32.1 UGS igb0After ~10 minutes, Routing table when WAN1 is down (pull coax from modem simulate ISP failure):
[2.4.5-RELEASE][root@pfSense.apt]/root: netstat -rnfinet Routing tables Internet: Destination Gateway Flags Netif Expire default 192.168.10.1 UGS igb1 1.0.0.1 192.168.100.1 UGHS igb0 1.0.0.1/32 192.168.100.1 UGS igb0 8.8.4.4 192.168.10.1 UGHS igb1 8.8.4.4/32 192.168.10.1 UGS igb1 10.0.8.0/24 10.0.8.2 UGS ovpns1 10.0.8.1 link#13 UHS lo0 10.0.8.2 link#13 UH ovpns1 10.0.9.0/24 10.0.9.2 UGS ovpns2 10.0.9.1 link#14 UHS lo0 10.0.9.2 link#14 UH ovpns2 127.0.0.1 link#6 UH lo0 192.168.1.0/24 link#10 U lagg1 192.168.1.1 link#10 UHS lo0 192.168.2.0/24 link#11 U lagg1.2 192.168.2.1 link#11 UHS lo0 192.168.3.0/24 link#12 U lagg1.3 192.168.3.1 link#12 UHS lo0 192.168.10.0/24 192.168.10.1 UGS igb1 192.168.10.23 link#2 UHS lo0 192.168.100.0/24 link#1 U igb0 192.168.100.1/32 192.168.100.1 UGS igb0 192.168.100.11 link#1 UHS lo0**** AT THIS POINT I CAN NOW PING 1.0.0.1 FROM MY CLIENS when only WAN2 is operational, althtough the behavior seems to "flap" accessible/inaccessible likely as the modem reboots ****
matthew@Inspiron13-7378 $ ping 1.0.0.1 PING 1.0.0.1 (1.0.0.1) 56(84) bytes of data. 64 bytes from 1.0.0.1: icmp_seq=1 ttl=52 time=37.1 ms 64 bytes from 1.0.0.1: icmp_seq=2 ttl=52 time=46.5 ms 64 bytes from 1.0.0.1: icmp_seq=3 ttl=52 time=37.4 ms 64 bytes from 1.0.0.1: icmp_seq=4 ttl=52 time=130 ms 64 bytes from 1.0.0.1: icmp_seq=5 ttl=52 time=354 ms 64 bytes from 1.0.0.1: icmp_seq=6 ttl=52 time=34.2 ms 64 bytes from 1.0.0.1: icmp_seq=7 ttl=52 time=40.1 msRouting table when WAN1 is down no-carrier (pull Ethernet between modem/router):
[2.4.5-RELEASE][root@pfSense.apt]/root: netstat -rnfinet Routing tables Internet: Destination Gateway Flags Netif Expire default 192.168.10.1 UGS igb1 8.8.4.4 192.168.10.1 UGHS igb1 8.8.4.4/32 192.168.10.1 UGS igb1 10.0.8.0/24 10.0.8.2 UGS ovpns1 10.0.8.1 link#13 UHS lo0 10.0.8.2 link#13 UH ovpns1 10.0.9.0/24 10.0.9.2 UGS ovpns2 10.0.9.1 link#14 UHS lo0 10.0.9.2 link#14 UH ovpns2 127.0.0.1 link#6 UH lo0 192.168.1.0/24 link#10 U lagg1 192.168.1.1 link#10 UHS lo0 192.168.2.0/24 link#11 U lagg1.2 192.168.2.1 link#11 UHS lo0 192.168.3.0/24 link#12 U lagg1.3 192.168.3.1 link#12 UHS lo0 192.168.10.0/24 192.168.10.1 UGS igb1 192.168.10.23 link#2 UHS lo0**** AT THIS POINT I CAN NOW PING 1.0.0.1 FROM MY CLIENS when only WAN2 is operational, and it seems very stable reachable ****
If it helps, here is a visual from my home automation server (which I have PING a bunch of stuff to get a quick high level picture of my entire network status)
-
@derelict said in Static routes...not as expected?:
@mmiller7 You don't have to do any of that and you are probably confusing something by doing so.
Setting 1.0.0.1 as the gateway monitoring address automatically creates the host route out that interface that you are trying to create manually.
Setting 1.0.0.1 as a DNS server with a gateway set (I can't see if you have a gateway set or not) does the same thing.
There is zero need to try to make a static route like that.
I'll try removing the static route entries and see if it changes anything in my test.
-
@derelict I've deleted my manually-entered static rules (but left the monitor IP and DNS); no change.
-
@mmiller7 said in Static routes...not as expected?:
@derelict I've deleted my manually-entered static rules (but left the monitor IP and DNS); no change.
Hello, did you solve the problem because I have the same.
thank you
-
@philippe-richard Not solved, there is a bug report mentioned earlier in the thread on it that from what I gather is not yet rolled into a release (states targeting CE-Next, which I gather is an upcoming release not yet finished).
As a workaround, I added firewall rules to "pass" the outbound traffic that should go on my static route thru the gateway I want, followed by rules to "block" the outbound traffic matching that for every other way it might get out. This is a messy kluge but at least kinda works around it.
I very much hope that they implement a full fix in an upcoming release, I'd love to get rid of all my extra convoluted rules.
-
@mmiller7 Hello, thank you for your answer. I think I'll do like you while waiting for an update that will fix this problem. I wish you a good day.