Dynamic DNS - failing to lookup WAN IP in 2.4.4
-
I just upgraded a box to 2.4.4, and now Dynamic DNS is failing to figure out the public IP of one of the two interfaces.
This machine has a cable modem with static IP on one WAN, and a secondary WAN (via OPT4) that has an RFC1918 IP from a DSL modem (that can't be put into passthrough mode for various reasons).
The Default Gateway on the box is pointing at the WAN interface (with the static IP) though.
In the past, Dynamic DNS was still able to figure out the DSL / OPT4 interface's "real" public address and update a DDNS system (in this case Route53), but now it is failing.
rc.dyndns.update: Dynamic DNS (actual-hostname.example.com) There was an error trying to determine the public IP for interface - opt4 (igb5 ).Not really sure what's wrong here but happy to provide any info needed to troubleshoot further. I was going to open an Issue on Redmine but wanted to throw it up here first in case anybody had suggestions.
When I try:
curl --interface igb5 http://checkip.dyndns.comI get basically nothing from it. Going back and adding a -v, I'm seeing:
* Rebuilt URL to: checkip.dyndns.com/ * Trying 162.88.96.194... * TCP_NODELAY set * Local Interface igb5 is ip 192.168.254.1 using address family 2 * Local port: 0 * connect to 162.88.96.194 port 80 failed: Operation timed out * Trying 162.88.100.200... * TCP_NODELAY set * Local Interface igb5 is ip 192.168.254.1 using address family 2 * Local port: 0As best I can tell the interface is functional though - gateway status shows that it is "up," and I have it set to ping a WAN IP (4.2.2.4) for gateway status since the "actual" gateway is just the DSL modem at the other end of a 3 foot piece of Cat5e.
-
@zprime I'm also seeing the same issue on 2.4.4 with HE dyndns updating not working.
-
Here's what I see from the System Log with "Verbose mode" enabled for this DDNS host:
Oct 11 00:30:30 php-cgi rc.dyndns.update: Dynamic DNS (myddnsname.example.com) There was an error trying to determine the public IP for interface - opt4 (igb5 ). Oct 11 00:30:30 php-cgi rc.dyndns.update: Dynamic DNS (myddnsname.example.com): running get_failover_interface for opt4. found igb5 Oct 11 00:30:00 php-cgi rc.dyndns.update: Dynamic DNS: updatedns() starting -
From the firewall, what does
route -n get 162.88.96.194show?What about a
tracerouteormtrto162.88.96.194? -
This post is deleted! -
route -n get 162.88.96.194 route to: 162.88.96.194 destination: 0.0.0.0 mask: 0.0.0.0 gateway: 70.60.xxx.yyy [remote end of a static /30 I don't want to provide] fib: 0 interface: igb1 [this is the other WAN interface, cable modem] flags: <UP,GATEWAY,DONE,STATIC> recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0The traceroute seems to be going through the cable modem as well, which makes sense given the route above. (Note that I have no idea why the first hop shows up as 192.168.0.1, I didn't change that - I think this is some weirdness with how Spectrum does static IPs on their gear, the modem seems to actually get the other end of the static /30):
traceroute -I 162.88.96.194 traceroute to 162.88.96.194 (162.88.96.194), 64 hops max, 48 byte packets 1 192.168.0.1 (192.168.0.1) 0.282 ms 0.195 ms 0.182 ms 2 142.254.157.45 (142.254.157.45) 8.361 ms 7.845 ms 7.980 ms 3 ae62.srsloh0401h.midwest.rr.com (24.164.114.9) 381.812 ms 25.320 ms 22.072 ms 4 agg31.mcdnohbg01r.midwest.rr.com (24.33.101.164) 9.971 ms 9.460 ms 9.291 ms 5 be14.pltsohae01r.midwest.rr.com (65.29.1.87) 16.021 ms 10.504 ms 16.120 ms 6 be25.clmkohpe01r.midwest.rr.com (65.29.1.28) 20.998 ms 21.365 ms 15.838 ms 7 107.14.17.252 (107.14.17.252) 28.415 ms 25.318 ms 23.973 ms 8 * lag-82.ear2.Chicago2.Level3.net (4.68.72.197) 24.497 ms 23.703 ms 9 ae-1-9.bar1.Warsaw1.Level3.net (4.69.153.70) 141.563 ms 141.331 ms 192.050 ms 10 dialup-212.162.18.138.frankfurt1.eu.level3.net (213.242.117.138) 139.285 ms 142.733 ms 138.591 ms 11 checkip1-waw.ct.as15135.net (162.88.96.194) 139.806 ms 138.877 ms 139.112 ms -
For what it is worth I am seeing very similar errors but have no problems with using curl to get my ip address when I point it to the correct wan interface. I also have a dual-wan setup with one wan connection connected to Spectrum, so maybe there's an issue with multi-wan setups here and default routes being set (or rather not set) properly.
Under System / Routing / Gateways my IPv4 gateway is set to be a gateway group (which otherwise appears to work.)
curl http://checkip.dyndns.com curl: (7) Couldn't connect to server curl --interface lagg0 http://checkip.dyndns.com <html><head><title>Current IP Check</title></head><body>Current IP Address: 74.*.*.*</body></html>Yet my logs also show the same error:
Oct 13 19:30:20 php-fpm 88624 /services_dyndns_edit.php: Dynamic DNS (460004) There was an error trying to determine the public IP for interface - wan (lagg0 ). Oct 13 19:30:20 php-fpm 88624 /services_dyndns_edit.php: Dynamic DNS (460004): running get_failover_interface for wan. found lagg0 Oct 13 19:30:20 php-fpm 88624 /services_dyndns_edit.php: Keep current gateway, its already part of the group members. Oct 13 19:30:20 php-fpm 88624 /services_dyndns_edit.php: Keep current gateway, its already part of the group members. Oct 13 19:30:20 php-fpm 88624 /services_dyndns_edit.php: Dynamic DNS: updatedns() starting Oct 13 19:30:20 php-fpm 88624 /services_dyndns_edit.php: End of configuration backup to https://acb.netgate.com/save (success). Oct 13 19:30:20 php-fpm 88624 /services_dyndns_edit.php: Beginning configuration backup to .https://acb.netgate.com/saveHere is some network debug output:
dig checkip.dyndns.com ; <<>> DiG 9.12.2-P1 <<>> checkip.dyndns.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46077 ;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;checkip.dyndns.com. IN A ;; ANSWER SECTION: checkip.dyndns.com. 59 IN A 162.88.100.200 checkip.dyndns.com. 59 IN A 216.146.38.70 checkip.dyndns.com. 59 IN A 216.146.43.70 checkip.dyndns.com. 59 IN A 216.146.43.71 checkip.dyndns.com. 59 IN A 162.88.96.194 route -n get 162.88.100.200 route: route has not been found route -n get 162.88.96.194 route: route has not been found traceroute 162.88.96.194 traceroute: findsaddr: failed to connect to peer for src addr selection. traceroute 162.88.100.200 traceroute: findsaddr: failed to connect to peer for src addr selection. traceroute -i lagg0 162.88.100.200 traceroute to 162.88.100.200 (162.88.100.200), 64 hops max, 40 byte packets 1 192.168.0.1 (192.168.0.1) 0.644 ms 0.441 ms 0.382 ms 2 * * * 3 agg42.nyclnyrg02h.nyc.rr.com (68.173.200.170) 13.320 ms 13.078 ms 13.675 ms 4 agg101.nyquny9101r.nyc.rr.com (68.173.198.34) 12.766 ms 18.789 ms 21.636 ms 5 bu-ether25.nycmny837aw-bcr00.tbone.rr.com (107.14.19.22) 23.067 ms 14.494 ms 20.392 ms 6 0.ae4.pr0.nyc20.tbone.rr.com (66.109.1.35) 13.486 ms 0.ae2.pr0.nyc20.tbone.rr.com (107.14.19.147) 15.363 ms 0.ae1.pr0.nyc20.tbone.rr.com (66.109.6.163) 11.467 ms 7 ix-ae-10-0.tcore1.n75-new-york.as6453.net (66.110.96.13) 12.685 ms ix-ae-6-0.tcore1.n75-new-york.as6453.net (66.110.96.53) 11.291 ms 9.900 ms 8 if-ae-12-2.tcore2.nto-new-york.as6453.net (66.110.96.6) 17.890 ms 24.259 ms 18.087 ms 9 if-ae-30-2.tcore1.aeq-ashburn.as6453.net (63.243.216.21) 21.579 ms 17.077 ms 18.224 ms 10 66.198.154.138 (66.198.154.138) 17.132 ms 17.535 ms 31.604 ms 11 checkip1-iad.ct.as15135.net (162.88.100.200) 13.761 ms 16.996 ms 15.450 ms traceroute -i lagg0 162.88.96.194 traceroute to 162.88.96.194 (162.88.96.194), 64 hops max, 40 byte packets 1 192.168.0.1 (192.168.0.1) 0.593 ms 0.515 ms 0.408 ms 2 * * * 3 agg42.nyclnyrg02h.nyc.rr.com (68.173.200.170) 13.654 ms 9.384 ms 8.415 ms 4 agg101.nyquny9101r.nyc.rr.com (68.173.198.34) 17.186 ms 14.809 ms 13.154 ms 5 bu-ether25.nycmny837aw-bcr00.tbone.rr.com (107.14.19.22) 10.177 ms bu-ether15.nycmny837aw-bcr00.tbone.rr.com (66.109.6.76) 17.182 ms 17.166 ms 6 0.ae1.pr0.nyc20.tbone.rr.com (66.109.6.163) 11.157 ms 0.ae4.pr0.nyc20.tbone.rr.com (66.109.1.35) 12.193 ms 0.ae0.pr0.nyc20.tbone.rr.com (66.109.6.157) 9.893 ms 7 ix-ae-10-0.tcore1.n75-new-york.as6453.net (66.110.96.13) 11.017 ms 12.742 ms ix-ae-6-0.tcore1.n75-new-york.as6453.net (66.110.96.53) 14.641 ms 8 if-ae-12-2.tcore2.nto-new-york.as6453.net (66.110.96.6) 115.171 ms 108.103 ms 117.912 ms 9 63.243.216.23 (63.243.216.23) 114.755 ms 110.493 ms 119.315 ms 10 if-ae-15-2.tcore2.l78-london.as6453.net (80.231.131.117) 106.245 ms 114.346 ms 113.293 ms 11 if-ae-14-2.tcore2.av2-amsterdam.as6453.net (80.231.131.161) 108.859 ms 105.252 ms 113.522 ms 12 if-ae-2-2.tcore1.av2-amsterdam.as6453.net (195.219.194.5) 118.261 ms 111.990 ms 114.518 ms 13 if-ae-21-2.thar1.w1t-warsaw.as6453.net (195.219.188.26) 110.869 ms 108.241 ms 104.174 ms 14 195.219.188.46 (195.219.188.46) 125.974 ms 116.079 ms 117.242 ms 15 checkip1-waw.ct.as15135.net (162.88.96.194) 128.644 ms 122.861 ms 115.071 ms -
Also it turns out that netstat -r shows no default route for ipv4.
netstat -r Routing tables Internet: Destination Gateway Flags Netif Expire dns.quad9.net 192.168.0.1 UGHS lagg0 localhost link#6 UH lo0 192.168.0.0/24 link#10 U lagg0 192.168.0.2 link#10 UHS lo0When I manually set the default route from the command line curl does work as expected.
/sbin/route add default 192.168.0.1 netstat -r Routing tables Internet: Destination Gateway Flags Netif Expire default 192.168.0.1 UGS lagg0 dns.quad9.net 192.168.0.1 UGHS lagg0 localhost link#6 UH lo0 192.168.0.0/24 link#10 U lagg0 192.168.0.2 link#10 UHS lo0 curl http://checkip.dyndns.com <html><head><title>Current IP Check</title></head><body>Current IP Address: 74.*.*.*</body></html>Unfortunately, the dyndns system still doesn't update the IP correctly, putting out the same error message as before.
The route to the dyndns servers do seem to show up correctly now though:
route -n get 162.88.100.200 route to: 162.88.100.200 destination: 0.0.0.0 mask: 0.0.0.0 gateway: 192.168.0.1 fib: 0 interface: lagg0 flags: <UP,GATEWAY,DONE,STATIC> recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0 route -n get 162.88.96.194 route to: 162.88.96.194 destination: 0.0.0.0 mask: 0.0.0.0 gateway: 192.168.0.1 fib: 0 interface: lagg0 flags: <UP,GATEWAY,DONE,STATIC> recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0 traceroute 162.88.96.194 traceroute to 162.88.96.194 (162.88.96.194), 64 hops max, 40 byte packets 1 192.168.0.1 (192.168.0.1) 0.720 ms 0.462 ms 0.319 ms 2 * * * 3 agg42.nyclnyrg02h.nyc.rr.com (68.173.200.170) 16.059 ms 17.995 ms 10.655 ms 4 agg101.nyquny9101r.nyc.rr.com (68.173.198.34) 17.502 ms 14.694 ms 27.919 ms 5 bu-ether15.nycmny837aw-bcr00.tbone.rr.com (66.109.6.76) 15.620 ms bu-ether25.nycmny837aw-bcr00.tbone.rr.com (107.14.19.22) 11.947 ms 16.948 ms 6 0.ae1.pr0.nyc20.tbone.rr.com (66.109.6.163) 11.540 ms 0.ae4.pr0.nyc20.tbone.rr.com (66.109.1.35) 12.186 ms 0.ae2.pr0.nyc20.tbone.rr.com (107.14.19.147) 10.108 ms 7 ix-ae-10-0.tcore1.n75-new-york.as6453.net (66.110.96.13) 11.691 ms ix-ae-6-0.tcore1.n75-new-york.as6453.net (66.110.96.53) 11.857 ms ix-ae-10-0.tcore1.n75-new-york.as6453.net (66.110.96.13) 12.705 ms 8 if-ae-12-2.tcore2.nto-new-york.as6453.net (66.110.96.6) 112.869 ms 111.025 ms 110.679 ms 9 63.243.216.23 (63.243.216.23) 121.514 ms 125.887 ms 114.568 ms 10 if-ae-15-2.tcore2.l78-london.as6453.net (80.231.131.117) 114.239 ms 103.748 ms 127.484 ms 11 if-ae-14-2.tcore2.av2-amsterdam.as6453.net (80.231.131.161) 117.696 ms 100.613 ms 112.804 ms 12 if-ae-2-2.tcore1.av2-amsterdam.as6453.net (195.219.194.5) 114.278 ms 119.824 ms 121.297 ms 13 if-ae-21-2.thar1.w1t-warsaw.as6453.net (195.219.188.26) 112.446 ms 110.462 ms 109.212 ms 14 195.219.188.46 (195.219.188.46) 118.848 ms 114.024 ms 119.057 ms 15 checkip1-waw.ct.as15135.net (162.88.96.194) 123.333 ms 122.574 ms 121.204 ms traceroute 162.88.100.200 traceroute to 162.88.100.200 (162.88.100.200), 64 hops max, 40 byte packets 1 192.168.0.1 (192.168.0.1) 0.732 ms 0.365 ms 0.268 ms 2 * * * 3 agg42.nyclnyrg02h.nyc.rr.com (68.173.200.170) 14.134 ms 8.922 ms 13.378 ms 4 agg101.nyquny9101r.nyc.rr.com (68.173.198.34) 19.033 ms 22.598 ms 20.827 ms 5 bu-ether25.nycmny837aw-bcr00.tbone.rr.com (107.14.19.22) 17.095 ms bu-ether15.nycmny837aw-bcr00.tbone.rr.com (66.109.6.76) 25.161 ms bu-ether25.nycmny837aw-bcr00.tbone.rr.com (107.14.19.22) 19.210 ms 6 0.ae0.pr0.nyc20.tbone.rr.com (66.109.6.157) 25.807 ms 8.018 ms 14.457 ms 7 ix-ae-6-0.tcore1.n75-new-york.as6453.net (66.110.96.53) 15.146 ms 9.237 ms 9.397 ms 8 if-ae-12-2.tcore2.nto-new-york.as6453.net (66.110.96.6) 19.758 ms 13.883 ms 16.811 ms 9 if-ae-30-2.tcore1.aeq-ashburn.as6453.net (63.243.216.21) 30.533 ms 18.850 ms 20.282 ms 10 66.198.154.138 (66.198.154.138) 20.389 ms 14.947 ms 16.206 ms 11 checkip1-iad.ct.as15135.net (162.88.100.200) 17.390 ms 17.407 ms 15.349 ms -
If you have no default route, that's a different problem and may be causing this. Make sure you have the correct gateway chosen for the default gateway under System > Routing. If you have it set to a gateway group, make sure that the members of the gateway group are set for failover (one gateway per tier).
It's not the first time I've seen someone mention that having a gateway group selected there resulted in not having a default route but I still haven't managed to make it happen here yet.
If it works when the default route is set then the problem is definitely the routing. But if it's not working when the default route it set, the cause still isn't quite clear. If the firewall can reach out, there isn't any reason it should be timing out like that, unless it's failing somewhere in between.
-
The default gateway group is set to failover with each member set to a different tier.
Hopefully something will turn up at some point...
-
@jimp On my installation, the default route is set to the cable modem static gateway, so no gateway group issue in play for me.
"Link#6" is igb5 which is my OPT4 interface going to the DSL modem. The modem has an IP of 192.168.254.254/24 (it's from Windstream), the pfSense box has .1, which we set statically.
"Link #2" is igb1 which is the static IP (/30) from Spectrum / Time Warner Business.
netstat -nr Routing tables Internet: Destination Gateway Flags Netif Expire default 70.60.x.y UGS igb1 1.0.0.1 192.168.254.254 UGHS igb5 1.1.1.1 70.60.x.y UGHS igb1 4.2.2.4 192.168.254.254 UGHS igb5 4.2.2.5 70.60.x.y UGHS igb1 8.8.4.4 192.168.254.254 UGHS igb5 8.8.8.8 70.60.x.y UGHS igb1 9.9.9.9 70.60.x.y UGHS igb1 10.17.0.0/24 link#1 U igb0 10.17.0.1 link#1 UHS lo0 10.254.254.0/24 10.254.254.2 UGS ovpns1 10.254.254.1 link#11 UHS lo0 10.254.254.2 link#11 UH ovpns1 70.60.x.w/30 link#2 U igb1 70.60.x.z link#2 UHS lo0 127.0.0.1 link#8 UH lo0 149.112.112.112 192.168.254.254 UGHS igb5 192.168.254.0/24 link#6 U igb5 192.168.254.1 link#6 UHS lo0ping -c 4 192.168.254.254 PING 192.168.254.254 (192.168.254.254): 56 data bytes 64 bytes from 192.168.254.254: icmp_seq=0 ttl=64 time=1.170 ms 64 bytes from 192.168.254.254: icmp_seq=1 ttl=64 time=1.071 ms 64 bytes from 192.168.254.254: icmp_seq=2 ttl=64 time=1.084 ms 64 bytes from 192.168.254.254: icmp_seq=3 ttl=64 time=1.083 ms --- 192.168.254.254 ping statistics --- 4 packets transmitted, 4 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 1.071/1.102/1.170/0.040 ms arp -i igb5 -a ? (192.168.254.254) at 4c:17:eb:21:26:09 on igb5 expires in 1178 seconds [ethernet] ? (192.168.254.1) at 00:08:a2:09:5a:15 on igb5 permanent [ethernet] traceroute -I -i igb5 checkip.dyndns.com traceroute: Warning: checkip.dyndns.com has multiple addresses; using 216.146.43.71 traceroute to checkip.dyndns.com (216.146.43.71), 64 hops max, 48 byte packets 1 * * * 2 * * * 3 * * * 4 * * * ^C ping -c 4 -S 192.168.254.1 checkip.dyndns.com PING checkip.dyndns.com (216.146.43.71) from 192.168.254.1: 56 data bytes --- checkip.dyndns.com ping statistics --- 4 packets transmitted, 0 packets received, 100.0% packet loss [This seems like it might be some of the problem...]It seems like I can't get a traceroute (ICMP) through the crappy DSL modem. However, apinger is not complaining about the connection being down, and it is set to ping 4.2.2.4 from that interface (instead of using the gateway IP). So maybe there's an apinger bug in play here and my connection is actually down but not correctly showing it?
It would be immensely more helpful if BSD ping could be forced to send from a specific interface... I'm wondering ifping -S <int_ip>is trying to send the traffic out the wrong interface somehow and is maybe a red herring?So let's play some more - adding a static route to 216.146.43.71 (which is one of the IPs for checkip.dyndns.com) to force it through the DSL gateway:
route add -host 216.146.43.71 192.168.254.254 add host 216.146.43.71: gateway 192.168.254.254 ping 216.146.43.71 PING 216.146.43.71 (216.146.43.71): 56 data bytes 64 bytes from 216.146.43.71: icmp_seq=0 ttl=49 time=149.241 ms 64 bytes from 216.146.43.71: icmp_seq=1 ttl=49 time=150.749 ms 64 bytes from 216.146.43.71: icmp_seq=2 ttl=49 time=151.432 ms ^C --- 216.146.43.71 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss traceroute -I 216.146.43.71 traceroute to 216.146.43.71 (216.146.43.71), 64 hops max, 48 byte packets 1 192.168.254.254 (192.168.254.254) 1.165 ms 0.912 ms 0.930 ms 2 h3.176.142.40.ip.windstream.net (40.142.176.3) 20.753 ms 20.375 ms 19.908 ms 3 ae2-0.agr03.hdsn01-oh.us.windstream.net (40.136.113.108) 21.559 ms 22.031 ms 21.936 ms 4 et9-0-0-0.cr01.cley01-oh.us.windstream.net (40.136.97.135) 24.981 ms 20.902 ms 24.131 ms 5 et11-0-0-0.cr01.chcg01-il.us.windstream.net (40.128.248.71) 27.746 ms 30.224 ms 30.609 ms 6 chi-b21-link.telia.net (80.239.194.41) 30.788 ms 32.021 ms 28.424 ms 7 nyk-bb3-link.telia.net (80.91.246.163) 147.362 ms 149.769 ms 147.627 ms 8 ldn-bb3-link.telia.net (62.115.135.95) 144.829 ms 144.190 ms 144.030 ms 9 hbg-bb1-link.telia.net (80.91.249.11) 140.047 ms 132.839 ms 130.322 ms 10 war-b1-link.telia.net (62.115.135.187) 145.479 ms 146.117 ms 145.660 ms 11 dnsnet-ic-320436-war-b1.c.telia.net (213.248.68.135) 151.281 ms 147.401 ms 151.799 ms 12 checkip.dyndns.com (216.146.43.71) 150.409 ms 152.881 ms 151.245 ms curl -v --interface igb5 http://216.146.43.71 * Rebuilt URL to: http://216.146.43.71/ * Trying 216.146.43.71... * TCP_NODELAY set * Local Interface igb5 is ip 192.168.254.1 using address family 2 * Local port: 0 * Connected to 216.146.43.71 (216.146.43.71) port 80 (#0) > GET / HTTP/1.1 > Host: 216.146.43.71 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Content-Type: text/html < Server: DynDNS-CheckIP/1.0.1 < Connection: close < Cache-Control: no-cache < Pragma: no-cache < Content-Length: 104 < <html><head><title>Current IP Check</title></head><body>Current IP Address: 75.90.aaa.bbb</body></html> * Closing connection 0 [!! WORKS !!]Something really weird is going on here. For whatever reason the traffic is not correctly egressing through the specified interface when using
curl --interface, and it's only going the way we want if I manually add a static route. I'm not exactly sure how the PHP code is hitting checkip.dyndns.com directly via a given interface, but something has changed behavior-wise that is making this fail. (Guessing maybe it's an underlying OS thing at this point, but I suppose it could also be something with curl if that is just being called via PHP?)
