NTP with time.google.com after a reboot
-
@Cylosoft said in NTP with time.google.com after a reboot:
allow IPv6" unchecked.
But was your wan actually getting an IPv6 address?
What do you have set for your wan interface?
If you have no IPv6 address, ntp should never try and talk to a Ipv6 - cuz yeah never going to work ;)
-
@johnpoz WAN IPv6 is set to none. I agree there is no way NTP could ever use IPv6. I think the auto setting is broken. If I restart the NTP service it works fine. But coming out of a reboot it always pulls IPv6 and stays unreach/pending until I restart NTP server. I tried this on a 2.7 and 23.x box.
-
@Cylosoft You could stop resolving AAAA that might prevent it..
Simple
private-address: ::/0
in your unbound config and AAAA will never get resolved, so did setting it to v4 fix it.. That was added a while ago to fix this specific issue I believe.
https://redmine.pfsense.org/issues/10322
I just point pfsense to my local ntp server running on a pi, so I have never seen this issue.
If you used the default ntp pool address, this would never happen either because there is no IPv6 in it, ntp pools only have IPv6 in specific pool address 2 I believe. I don't think any of the pfsense.ntp.pool.org will return Ipv6..
;; QUESTION SECTION: ;2.pool.ntp.org. IN AAAA ;; ANSWER SECTION: 2.pool.ntp.org. 3600 IN AAAA 2620:2d:4000:1::3f 2.pool.ntp.org. 3600 IN AAAA 2620:149:a33:3000::1e2 2.pool.ntp.org. 3600 IN AAAA 2001:470:8d7e:400:20c:29ff:fe86:4251 2.pool.ntp.org. 3600 IN AAAA 2607:7c80:54:3::32$ dig 0.pool.ntp.org AAAA ; <<>> DiG 9.16.41 <<>> 0.pool.ntp.org AAAA ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30482 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;0.pool.ntp.org. IN AAAAAn intelligent man is sometimes forced to be drunk to spend time with his fools
If you get confused: Listen to the Music Play
Please don't Chat/PM me for help, unless mod related
SG-4860 24.11 | Lab VMs 2.7.2, 24.11 -
@johnpoz Yes IPv4 setting on NTP fixed everything.
-
@Cylosoft great to hear, but yeah 2.pfsense.pool.ntp.org does return AAAA
;; QUESTION SECTION: ;2.pfsense.pool.ntp.org. IN AAAA ;; ANSWER SECTION: 2.pfsense.pool.ntp.org. 3600 IN AAAA 2600:3c01::f03c:92ff:fe8c:f7c9 2.pfsense.pool.ntp.org. 3600 IN AAAA 2620:149:a32:3000::1f2 2.pfsense.pool.ntp.org. 3600 IN AAAA 2607:f1c0:1801:61::1 2.pfsense.pool.ntp.org. 3600 IN AAAA 2606:4700:f1::1but 0, 1 or 3 do not.
-
@johnpoz We typically put time.google.com and time.cloudflare.com. In our firewall life previous to PF we had issues with some ntp.org pools and had to switch off. Even though it makes no sense we just still don't use them.
-
@Cylosoft if you have no IPv6 - you might want to use the unbound thing so no AAAA ever get resolved.. Glad the v4 ntp setting worked for you.
-
@johnpoz thanks I wanted to share the results after with a dns lookup or dig with Google
private-address: ::/0
Added to DNS resolver advanced config.
-
Re: NTP with time.google.com after a reboot
[Sorry for the cross post; I mistakenly used reply to topic but this forum software won’t let me delete my own erroneous post.]
This happened to me as well. I have ipv6 disabled. NTP configured with auto dns was fine until a reboot. It then started trying to open an ipv6 socket on my Starlink interface and failing every minute or so.
I use pool.ntp.org. I’m running pfSense v23.05.1-RELEASE. I also have a T-mobile interface in load balancing configuration.
I’ve experienced this in earlier versions as well.
I set up ipv6 on the starlink interface and ntp stopped generating the error. After finding the above I also set ntp dns to ipv4. I may someday go back and disable IPv6 on the Starlink interface but it doesn’t seem to matter much either way.
It appears ntp auto function has a bug where in some circumstances it attempts to open an IPv6 socket one some external interfaces even when they don’t have IPv6 configured and IPv6 is disabled system wide.
This bug doesn’t seem to interfere with ntp working properly but it does generate thousands of errors. In the interest of making pfSense as bug free as possible this should be investigated and fixed.
-
@Mission-Ghost have you tried to use a NAT port forwarding rule? This will redirect all requests for NTP to the server of your choice?
This is what I do as I had many devices requesting NTP all over.
Lan interface -> anything source address inside the Lan net -> over UDP-> any source port -> redirect anything not going to the firewall or loopbacks (any other ntp requests) -> that wants NTP as its destination port
This will redirect any NTP requests that are not being sent to the firewall right back to it. Port forwarding.
(alias has the firewalls address and its loopback, you would need to make 2 one for IPv6 and one rule for IPv4)
-
@JonathanLee thanks for your post.
Yes, my NTP rules are configured to redirect all of my LAN NTP requests to the router's NTP server. Our LAN devices keep excellent time using this.
However, it doesn't seem to help with or be a factor in the reboot-then-IPv6-socket request error after reboot since NTP configuration.
-
@Mission-Ghost if it happens on boot - you might want to take a look here
https://docs.netgate.com/pfsense/en/latest/services/ntpd/bootstrap.html
I changed it on mine just recently so it points to my local ntp server vs google time
An intelligent man is sometimes forced to be drunk to spend time with his fools
If you get confused: Listen to the Music Play
Please don't Chat/PM me for help, unless mod related
SG-4860 24.11 | Lab VMs 2.7.2, 24.11 -
@johnpoz I wrote up a detailed reply with my troubleshooting and results following your suggestion, but I can't post it. I get a message that the post contains spam courtesy of some (defective) spam filter service. How do I work around this?
-
@johnpoz thanks for the idea.
I set the conf file with ip addresses of the ntp.org servers plus one time.gov server, disabled the Starlink IPv6 configuration and left the NTP server DNS setting to IPv4 only and rebooted.
The error returned and repeats as before. It looks like this:
Jul 7 13:43:12 ntpd 46651 failed to init interface for address fe80::f2ad:4eff:fe1d:bd1f%12 Jul 7 13:43:12 ntpd 46651 unable to create socket on mvneta0.4092 (31) for fe80::f2ad:4eff:fe1d:bd1f%12#123 ... Jul 7 13:22:51 ntpd 46651 0.0.0.0 c615 05 clock_sync Jul 7 13:22:51 ntpd 46651 65.100.46.166 101a 8a sys_peer Jul 7 13:22:49 ntpd 46651 162.159.200.1 0014 84 reachable . . . [Bootup and initial NTP sequence...]So, I've now gone back and re-configured the Starlink interface in accordance with Step 1 in the following procedure as of about 1:52pm:
https://starlinkmag.com/starlink-support-ipv6/
IPv6 remains disabled in the rest of the router's configuration.
1:52pm was about when the next IPv6 failure was due to show up in the log.
It may be meaningful that the retry is exactly every five minutes, like clockwork, to the second.
Returning to the NTP log, it is now 1:57pm and still no more IPv6 error:
Jul 7 13:57:30 ntpd 46651 65.100.46.166 143a 8a sys_peer Jul 7 13:48:12 ntpd 46651 failed to init interface for address fe80::f2ad:4eff:fe1d:bd1f%12 Jul 7 13:48:12 ntpd 46651 unable to create socket on mvneta0.4092 (32) for fe80::f2ad:4eff:fe1d:bd1f%12#123 ... Jul 7 13:22:51 ntpd 46651 0.0.0.0 c615 05 clock_sync Jul 7 13:22:51 ntpd 46651 65.100.46.166 101a 8a sys_peer . . .So, in my configuration (on a Netgate 1100), I'm concluding so far:
- The error only points to the Starlink WAN interface. No error shows for the T-Mobile WAN interface.
- Changing the NTP server DNS resolution to IPv4 only fails to eliminate the error.
- Creating a /conf/ntp-boot-time-servers file with IPv4 address pointers to various ntp.org and time.gov servers fails to eliminate the error.
- Enabling IPv6 on the Starlink interface eliminates the error.
-
@Mission-Ghost Looks like one can't post more than a few lines of pfSense logs before the forum spam filter misinterprets it as something spammy. So I clipped the log output and it posted without interference from the spam filter.
-
@johnpoz I was just going to quote that as I just leaned this came back to this post but you beat me to it
"The firewall performs a one-time sync to multiple NTP servers with static IP addresses from Google Public NTP. This avoids a chicken-and-egg problem where the firewall cannot resolve NTP servers because DNSSEC, which is enabled by default, cannot function when the clock is inaccurate. This happens much later in the boot process because it cannot be performed until after the firewall has configured its interfaces and routing.
This gets the clock as close to accurate as possible without a persistent NTP daemon. There is a hard timeout of 30 seconds in case the upstream servers are unreachable.
Changing Clock Bootstrap Behavior
The NTP clock bootstrap behavior can easily be disabled or changed if an administrator does not want a firewall to contact the default list of servers.To disable the bootstrap, create the file /conf/ntp-boot-time-servers as an empty file. If the file exists and is empty the firewall will skip the initial sync.
To use alternate servers, create the file /conf/ntp-boot-time-servers and add in one or more IP addresses separated by a single space each. If this file contains a list of space-separated IP addresses, the firewall will use those for the bootstrap sync instead" (Netgate docs).
-
@JonathanLee I tried creating the file and pointing it to ntp.org and time.gov NTP server IP addresses, but to no effect at preventing the attempts to use a non-existent IPv6 WAN interface address. I'll try an empty file and see if that has an effect and report back.
Even so, I think an issue exists where the NTP server fails to respect the disablement of IPv6 and then infinitely loops trying an IPv6 WAN interface address every five minutes, even if there isn't one.
-
@johnpoz ok, I also tried an empty /conf/ntp-boot-time-servers file to disable the bootstrap setting of time, disabled IPv6 configuration on the Starlink interface rebooted, and NTP still attempted to use an IPv6 interface address on the Starlink interface, even though none exists:
Jul 8 15:43:59 ntpd 81262 failed to init interface for address fe80::f2ad:4eff:fe1d:bd1f%12 Jul 8 15:43:59 ntpd 81262 unable to create socket on mvneta0.4092 (27) for fe80::f2ad:4eff:fe1d:bd1f%12#123 Jul 8 15:43:59 ntpd 81262 bind(45) AF_INET6 fe80::f2ad:4eff:fe1d:bd1f%12#123 flags 0x11 failed: Can't assign requested address Jul 8 15:43:40 ntpd 81262 108.61.56.35 144a 8a sys_peerI re-enabled the Starlink IPv6 setup, deleted the /conf/ntp-boot-time-servers file (restoring default behavior) and NTP is now still reporting being unable to to create a socket on the Starlink interface IPv6 address. I must be missing something I'd done before. To wit:
Jul 8 16:11:20 ntpd 78011 failed to init interface for address fe80::f2ad:4eff:fe1d:bd1f%18 Jul 8 16:11:20 ntpd 78011 unable to create socket on mvneta0.4092 (30) for fe80::f2ad:4eff:fe1d:bd1f%18#123 Jul 8 16:11:20 ntpd 78011 bind(45) AF_INET6 fe80::f2ad:4eff:fe1d:bd1f%18#123 flags 0x11 failed: Can't assign requested addressBarring documentation that this attempt to use an IPv6 interface for NTP socket creation when IPv6 is disabled on the router is normal behavior, this appears to be a bug. Otherwise, it's hard for me to figure out why normal behavior would be to attempt the use of an IPv6 address when it's disabled system wide and loop an error every five minutes forever.
Interestingly, earlier in the NTP log during boot, it appears to be trying to use IPv6 addresses on every interface, but only fails on the Starlink interface:
Jul 8 15:55:47 ntpd 78011 failed to init interface for address fe80::f2ad:4eff:fe1d:bd1f%18 Jul 8 15:55:47 ntpd 78011 unable to create socket on mvneta0.4092 (23) for fe80::f2ad:4eff:fe1d:bd1f%18#123 Jul 8 15:55:47 ntpd 78011 bind(43) AF_INET6 fe80::f2ad:4eff:fe1d:bd1f%18#123 flags 0x11 failed: Can't assign requested address Jul 8 15:55:47 ntpd 78011 Listen normally on 22 mvneta0.60 192.168.60.1:123 Jul 8 15:55:47 ntpd 78011 Listen normally on 21 mvneta0.60 [fe80::f2ad:4eff:fe1d:bd1f%17]:123 Jul 8 15:55:47 ntpd 78011 Listen normally on 20 mvneta0.50 192.168.50.1:123 Jul 8 15:55:47 ntpd 78011 Listen normally on 19 mvneta0.50 [fe80::f2ad:4eff:fe1d:bd1f%16]:123 Jul 8 15:55:47 ntpd 78011 Listen normally on 18 mvneta0.40 192.168.40.1:123 Jul 8 15:55:47 ntpd 78011 Listen normally on 17 mvneta0.40 [fe80::f2ad:4eff:fe1d:bd1f%15]:123 Jul 8 15:55:47 ntpd 78011 Listen normally on 16 mvneta0.30 192.168.30.1:123 Jul 8 15:55:47 ntpd 78011 Listen normally on 15 mvneta0.30 [fe80::f2ad:4eff:fe1d:bd1f%14]:123 Jul 8 15:55:47 ntpd 78011 Listen normally on 14 mvneta0.20 192.168.20.1:123 Jul 8 15:55:47 ntpd 78011 Listen normally on 13 mvneta0.20 [fe80::f2ad:4eff:fe1d:bd1f%13]:123 Jul 8 15:55:47 ntpd 78011 Listen normally on 12 mvneta0.10 192.168.10.1:123 Jul 8 15:55:47 ntpd 78011 Listen normally on 11 mvneta0.10 [fe80::f2ad:4eff:fe1d:bd1f%12]:123 Jul 8 15:55:47 ntpd 78011 Listen normally on 10 mvneta0.4091 192.168.2.1:123 Jul 8 15:55:47 ntpd 78011 Listen normally on 9 mvneta0.4091 [fe80::f2ad:4eff:fe1d:bd1f%11]:123 Jul 8 15:55:47 ntpd 78011 Listen normally on 8 mvneta0.4090 192.168.12.216:123 Jul 8 15:55:47 ntpd 78011 Listen normally on 7 mvneta0.4090 [fe80::f2ad:4eff:fe1d:bd1f%10]:123 Jul 8 15:55:47 ntpd 78011 Listen normally on 6 lo0 10.10.10.1:123 Jul 8 15:55:47 ntpd 78011 Listen normally on 5 lo0 127.0.0.1:123 Jul 8 15:55:47 ntpd 78011 Listen normally on 4 lo0 [fe80::1%7]:123 Jul 8 15:55:47 ntpd 78011 Listen normally on 3 lo0 [::1]:123 Jul 8 15:55:47 ntpd 78011 Listen normally on 2 mvneta0 [fe80::f2ad:4eff:fe1d:bd1f%1]:123 Jul 8 15:55:47 ntpd 78011 Listen and drop on 1 v4wildcard 0.0.0.0:123 Jul 8 15:55:47 ntpd 78011 Listen and drop on 0 v6wildcard [::]:123 Jul 8 15:55:47 ntpd 78011 gps base set to 2023-01-01 (week 2243) Jul 8 15:55:47 ntpd 78011 basedate set to 2022-12-26 Jul 8 15:55:47 ntpd 78011 proto: precision = 0.320 usec (-21) Jul 8 15:55:47 ntpd 77912 ---------------------------------------------------- Jul 8 15:55:47 ntpd 77912 available at https://www.nwtime.org/support Jul 8 15:55:47 ntpd 77912 corporation. Support and training for ntp-4 are Jul 8 15:55:47 ntpd 77912 Inc. (NTF), a non-profit 501(c)(3) public-benefit Jul 8 15:55:47 ntpd 77912 ntp-4 is maintained by Network Time Foundation, Jul 8 15:55:47 ntpd 77912 ---------------------------------------------------- Jul 8 15:55:47 ntpd 77912 Command line: /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid Jul 8 15:55:47 ntpd 77912 ntpd 4.2.8p15@1.3728-o Sat Jan 7 19:55:07 UTC 2023 (1): Starting -
Ok, without rebooting I went back to the Starlink interface, disabled IPv6 again, then re-enabled it again and the NTP IPv6 socket error has stopped. I can't reboot now to test if the error returns with a reboot but will try later.
Seems flaky.
