NTP with time.google.com after a reboot

johnpoz · Jun 30, 2023, 11:59 AM

@Cylosoft You could stop resolving AAAA that might prevent it..

Simple

private-address: ::/0

in your unbound config and AAAA will never get resolved, so did setting it to v4 fix it.. That was added a while ago to fix this specific issue I believe.

https://redmine.pfsense.org/issues/10322

I just point pfsense to my local ntp server running on a pi, so I have never seen this issue.

If you used the default ntp pool address, this would never happen either because there is no IPv6 in it, ntp pools only have IPv6 in specific pool address 2 I believe. I don't think any of the pfsense.ntp.pool.org will return Ipv6..

;; QUESTION SECTION:
;2.pool.ntp.org.                        IN      AAAA

;; ANSWER SECTION:
2.pool.ntp.org.         3600    IN      AAAA    2620:2d:4000:1::3f
2.pool.ntp.org.         3600    IN      AAAA    2620:149:a33:3000::1e2
2.pool.ntp.org.         3600    IN      AAAA    2001:470:8d7e:400:20c:29ff:fe86:4251
2.pool.ntp.org.         3600    IN      AAAA    2607:7c80:54:3::32

$ dig 0.pool.ntp.org AAAA

; <<>> DiG 9.16.41 <<>> 0.pool.ntp.org AAAA
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30482
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;0.pool.ntp.org.                        IN      AAAA

Cylosoft · Jun 30, 2023, 11:55 AM

@johnpoz Yes IPv4 setting on NTP fixed everything.

johnpoz · Jun 30, 2023, 12:01 PM

@Cylosoft great to hear, but yeah 2.pfsense.pool.ntp.org does return AAAA

;; QUESTION SECTION:
;2.pfsense.pool.ntp.org.                IN      AAAA

;; ANSWER SECTION:
2.pfsense.pool.ntp.org. 3600    IN      AAAA    2600:3c01::f03c:92ff:fe8c:f7c9
2.pfsense.pool.ntp.org. 3600    IN      AAAA    2620:149:a32:3000::1f2
2.pfsense.pool.ntp.org. 3600    IN      AAAA    2607:f1c0:1801:61::1
2.pfsense.pool.ntp.org. 3600    IN      AAAA    2606:4700:f1::1

but 0, 1 or 3 do not.

Cylosoft · Jun 30, 2023, 12:09 PM

@johnpoz We typically put time.google.com and time.cloudflare.com. In our firewall life previous to PF we had issues with some ntp.org pools and had to switch off. Even though it makes no sense we just still don't use them.

johnpoz · Jun 30, 2023, 12:21 PM

@Cylosoft if you have no IPv6 - you might want to use the unbound thing so no AAAA ever get resolved.. Glad the v4 ntp setting worked for you.

JonathanLee · Jun 30, 2023, 3:30 PM

@johnpoz thanks I wanted to share the results after with a dns lookup or dig with Google

private-address: ::/0
Added to DNS resolver advanced config.

Mission-Ghost · Jul 7, 2023, 3:57 PM

Re: NTP with time.google.com after a reboot

[Sorry for the cross post; I mistakenly used reply to topic but this forum software won’t let me delete my own erroneous post.]

This happened to me as well. I have ipv6 disabled. NTP configured with auto dns was fine until a reboot. It then started trying to open an ipv6 socket on my Starlink interface and failing every minute or so.

I use pool.ntp.org. I’m running pfSense v23.05.1-RELEASE. I also have a T-mobile interface in load balancing configuration.

I’ve experienced this in earlier versions as well.

I set up ipv6 on the starlink interface and ntp stopped generating the error. After finding the above I also set ntp dns to ipv4. I may someday go back and disable IPv6 on the Starlink interface but it doesn’t seem to matter much either way.

It appears ntp auto function has a bug where in some circumstances it attempts to open an IPv6 socket one some external interfaces even when they don’t have IPv6 configured and IPv6 is disabled system wide.

This bug doesn’t seem to interfere with ntp working properly but it does generate thousands of errors. In the interest of making pfSense as bug free as possible this should be investigated and fixed.

JonathanLee · Jul 7, 2023, 4:07 PM

@Mission-Ghost have you tried to use a NAT port forwarding rule? This will redirect all requests for NTP to the server of your choice?

This is what I do as I had many devices requesting NTP all over.

Lan interface -> anything source address inside the Lan net -> over UDP-> any source port -> redirect anything not going to the firewall or loopbacks (any other ntp requests) -> that wants NTP as its destination port

This will redirect any NTP requests that are not being sent to the firewall right back to it. Port forwarding.

(alias has the firewalls address and its loopback, you would need to make 2 one for IPv6 and one rule for IPv4)

Mission-Ghost · Jul 7, 2023, 5:47 PM

@JonathanLee thanks for your post.

Yes, my NTP rules are configured to redirect all of my LAN NTP requests to the router's NTP server. Our LAN devices keep excellent time using this.

However, it doesn't seem to help with or be a factor in the reboot-then-IPv6-socket request error after reboot since NTP configuration.

johnpoz · Jul 7, 2023, 6:53 PM

@Mission-Ghost if it happens on boot - you might want to take a look here

https://docs.netgate.com/pfsense/en/latest/services/ntpd/bootstrap.html

I changed it on mine just recently so it points to my local ntp server vs google time

Mission-Ghost · Jul 7, 2023, 8:08 PM

@johnpoz I wrote up a detailed reply with my troubleshooting and results following your suggestion, but I can't post it. I get a message that the post contains spam courtesy of some (defective) spam filter service. How do I work around this?

Mission-Ghost · Jul 7, 2023, 9:39 PM

@johnpoz thanks for the idea.

I set the conf file with ip addresses of the ntp.org servers plus one time.gov server, disabled the Starlink IPv6 configuration and left the NTP server DNS setting to IPv4 only and rebooted.

The error returned and repeats as before. It looks like this:

Jul 7 13:43:12 	ntpd 	46651 	failed to init interface for address fe80::f2ad:4eff:fe1d:bd1f%12
Jul 7 13:43:12 	ntpd 	46651 	unable to create socket on mvneta0.4092 (31) for fe80::f2ad:4eff:fe1d:bd1f%12#123
...
Jul 7 13:22:51 	ntpd 	46651 	0.0.0.0 c615 05 clock_sync
Jul 7 13:22:51 	ntpd 	46651 	65.100.46.166 101a 8a sys_peer
Jul 7 13:22:49 	ntpd 	46651 	162.159.200.1 0014 84 reachable 
.
.
.
[Bootup and initial NTP sequence...]

So, I've now gone back and re-configured the Starlink interface in accordance with Step 1 in the following procedure as of about 1:52pm:

https://starlinkmag.com/starlink-support-ipv6/

IPv6 remains disabled in the rest of the router's configuration.

1:52pm was about when the next IPv6 failure was due to show up in the log.

It may be meaningful that the retry is exactly every five minutes, like clockwork, to the second.

Returning to the NTP log, it is now 1:57pm and still no more IPv6 error:

Jul 7 13:57:30 	ntpd 	46651 	65.100.46.166 143a 8a sys_peer
Jul 7 13:48:12 	ntpd 	46651 	failed to init interface for address fe80::f2ad:4eff:fe1d:bd1f%12
Jul 7 13:48:12 	ntpd 	46651 	unable to create socket on mvneta0.4092 (32) for fe80::f2ad:4eff:fe1d:bd1f%12#123
...
Jul 7 13:22:51 	ntpd 	46651 	0.0.0.0 c615 05 clock_sync
Jul 7 13:22:51 	ntpd 	46651 	65.100.46.166 101a 8a sys_peer 
.
.
.

So, in my configuration (on a Netgate 1100), I'm concluding so far:

The error only points to the Starlink WAN interface. No error shows for the T-Mobile WAN interface.
Changing the NTP server DNS resolution to IPv4 only fails to eliminate the error.
Creating a /conf/ntp-boot-time-servers file with IPv4 address pointers to various ntp.org and time.gov servers fails to eliminate the error.
Enabling IPv6 on the Starlink interface eliminates the error.

Mission-Ghost · Jul 7, 2023, 9:41 PM

@Mission-Ghost Looks like one can't post more than a few lines of pfSense logs before the forum spam filter misinterprets it as something spammy. So I clipped the log output and it posted without interference from the spam filter.

JonathanLee · Jul 8, 2023, 7:23 AM

@johnpoz I was just going to quote that as I just leaned this came back to this post but you beat me to it

"The firewall performs a one-time sync to multiple NTP servers with static IP addresses from Google Public NTP. This avoids a chicken-and-egg problem where the firewall cannot resolve NTP servers because DNSSEC, which is enabled by default, cannot function when the clock is inaccurate. This happens much later in the boot process because it cannot be performed until after the firewall has configured its interfaces and routing.

This gets the clock as close to accurate as possible without a persistent NTP daemon. There is a hard timeout of 30 seconds in case the upstream servers are unreachable.

Changing Clock Bootstrap Behavior
The NTP clock bootstrap behavior can easily be disabled or changed if an administrator does not want a firewall to contact the default list of servers.

To disable the bootstrap, create the file /conf/ntp-boot-time-servers as an empty file. If the file exists and is empty the firewall will skip the initial sync.

To use alternate servers, create the file /conf/ntp-boot-time-servers and add in one or more IP addresses separated by a single space each. If this file contains a list of space-separated IP addresses, the firewall will use those for the bootstrap sync instead" (Netgate docs).

Mission-Ghost · Jul 8, 2023, 9:33 PM

@JonathanLee I tried creating the file and pointing it to ntp.org and time.gov NTP server IP addresses, but to no effect at preventing the attempts to use a non-existent IPv6 WAN interface address. I'll try an empty file and see if that has an effect and report back.

Even so, I think an issue exists where the NTP server fails to respect the disablement of IPv6 and then infinitely loops trying an IPv6 WAN interface address every five minutes, even if there isn't one.

Mission-Ghost · Jul 8, 2023, 9:51 PM

@johnpoz ok, I also tried an empty /conf/ntp-boot-time-servers file to disable the bootstrap setting of time, disabled IPv6 configuration on the Starlink interface rebooted, and NTP still attempted to use an IPv6 interface address on the Starlink interface, even though none exists:

Jul 8 15:43:59 	ntpd 	81262 	failed to init interface for address fe80::f2ad:4eff:fe1d:bd1f%12
Jul 8 15:43:59 	ntpd 	81262 	unable to create socket on mvneta0.4092 (27) for fe80::f2ad:4eff:fe1d:bd1f%12#123
Jul 8 15:43:59 	ntpd 	81262 	bind(45) AF_INET6 fe80::f2ad:4eff:fe1d:bd1f%12#123 flags 0x11 failed: Can't assign requested address
Jul 8 15:43:40 	ntpd 	81262 	108.61.56.35 144a 8a sys_peer

I re-enabled the Starlink IPv6 setup, deleted the /conf/ntp-boot-time-servers file (restoring default behavior) and NTP is now still reporting being unable to to create a socket on the Starlink interface IPv6 address. I must be missing something I'd done before. To wit:

Jul 8 16:11:20 	ntpd 	78011 	failed to init interface for address fe80::f2ad:4eff:fe1d:bd1f%18
Jul 8 16:11:20 	ntpd 	78011 	unable to create socket on mvneta0.4092 (30) for fe80::f2ad:4eff:fe1d:bd1f%18#123
Jul 8 16:11:20 	ntpd 	78011 	bind(45) AF_INET6 fe80::f2ad:4eff:fe1d:bd1f%18#123 flags 0x11 failed: Can't assign requested address

Barring documentation that this attempt to use an IPv6 interface for NTP socket creation when IPv6 is disabled on the router is normal behavior, this appears to be a bug. Otherwise, it's hard for me to figure out why normal behavior would be to attempt the use of an IPv6 address when it's disabled system wide and loop an error every five minutes forever.

Interestingly, earlier in the NTP log during boot, it appears to be trying to use IPv6 addresses on every interface, but only fails on the Starlink interface:

Jul 8 15:55:47 	ntpd 	78011 	failed to init interface for address fe80::f2ad:4eff:fe1d:bd1f%18
Jul 8 15:55:47 	ntpd 	78011 	unable to create socket on mvneta0.4092 (23) for fe80::f2ad:4eff:fe1d:bd1f%18#123
Jul 8 15:55:47 	ntpd 	78011 	bind(43) AF_INET6 fe80::f2ad:4eff:fe1d:bd1f%18#123 flags 0x11 failed: Can't assign requested address
Jul 8 15:55:47 	ntpd 	78011 	Listen normally on 22 mvneta0.60 192.168.60.1:123
Jul 8 15:55:47 	ntpd 	78011 	Listen normally on 21 mvneta0.60 [fe80::f2ad:4eff:fe1d:bd1f%17]:123
Jul 8 15:55:47 	ntpd 	78011 	Listen normally on 20 mvneta0.50 192.168.50.1:123
Jul 8 15:55:47 	ntpd 	78011 	Listen normally on 19 mvneta0.50 [fe80::f2ad:4eff:fe1d:bd1f%16]:123
Jul 8 15:55:47 	ntpd 	78011 	Listen normally on 18 mvneta0.40 192.168.40.1:123
Jul 8 15:55:47 	ntpd 	78011 	Listen normally on 17 mvneta0.40 [fe80::f2ad:4eff:fe1d:bd1f%15]:123
Jul 8 15:55:47 	ntpd 	78011 	Listen normally on 16 mvneta0.30 192.168.30.1:123
Jul 8 15:55:47 	ntpd 	78011 	Listen normally on 15 mvneta0.30 [fe80::f2ad:4eff:fe1d:bd1f%14]:123
Jul 8 15:55:47 	ntpd 	78011 	Listen normally on 14 mvneta0.20 192.168.20.1:123
Jul 8 15:55:47 	ntpd 	78011 	Listen normally on 13 mvneta0.20 [fe80::f2ad:4eff:fe1d:bd1f%13]:123
Jul 8 15:55:47 	ntpd 	78011 	Listen normally on 12 mvneta0.10 192.168.10.1:123
Jul 8 15:55:47 	ntpd 	78011 	Listen normally on 11 mvneta0.10 [fe80::f2ad:4eff:fe1d:bd1f%12]:123
Jul 8 15:55:47 	ntpd 	78011 	Listen normally on 10 mvneta0.4091 192.168.2.1:123
Jul 8 15:55:47 	ntpd 	78011 	Listen normally on 9 mvneta0.4091 [fe80::f2ad:4eff:fe1d:bd1f%11]:123
Jul 8 15:55:47 	ntpd 	78011 	Listen normally on 8 mvneta0.4090 192.168.12.216:123
Jul 8 15:55:47 	ntpd 	78011 	Listen normally on 7 mvneta0.4090 [fe80::f2ad:4eff:fe1d:bd1f%10]:123
Jul 8 15:55:47 	ntpd 	78011 	Listen normally on 6 lo0 10.10.10.1:123
Jul 8 15:55:47 	ntpd 	78011 	Listen normally on 5 lo0 127.0.0.1:123
Jul 8 15:55:47 	ntpd 	78011 	Listen normally on 4 lo0 [fe80::1%7]:123
Jul 8 15:55:47 	ntpd 	78011 	Listen normally on 3 lo0 [::1]:123
Jul 8 15:55:47 	ntpd 	78011 	Listen normally on 2 mvneta0 [fe80::f2ad:4eff:fe1d:bd1f%1]:123
Jul 8 15:55:47 	ntpd 	78011 	Listen and drop on 1 v4wildcard 0.0.0.0:123
Jul 8 15:55:47 	ntpd 	78011 	Listen and drop on 0 v6wildcard [::]:123
Jul 8 15:55:47 	ntpd 	78011 	gps base set to 2023-01-01 (week 2243)
Jul 8 15:55:47 	ntpd 	78011 	basedate set to 2022-12-26
Jul 8 15:55:47 	ntpd 	78011 	proto: precision = 0.320 usec (-21)
Jul 8 15:55:47 	ntpd 	77912 	----------------------------------------------------
Jul 8 15:55:47 	ntpd 	77912 	available at https://www.nwtime.org/support
Jul 8 15:55:47 	ntpd 	77912 	corporation. Support and training for ntp-4 are
Jul 8 15:55:47 	ntpd 	77912 	Inc. (NTF), a non-profit 501(c)(3) public-benefit
Jul 8 15:55:47 	ntpd 	77912 	ntp-4 is maintained by Network Time Foundation,
Jul 8 15:55:47 	ntpd 	77912 	----------------------------------------------------
Jul 8 15:55:47 	ntpd 	77912 	Command line: /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid
Jul 8 15:55:47 	ntpd 	77912 	ntpd 4.2.8p15@1.3728-o Sat Jan 7 19:55:07 UTC 2023 (1): Starting

Mission-Ghost · Jul 8, 2023, 10:32 PM

Ok, without rebooting I went back to the Starlink interface, disabled IPv6 again, then re-enabled it again and the NTP IPv6 socket error has stopped. I can't reboot now to test if the error returns with a reboot but will try later.

Seems flaky.