NTP with time.google.com after a reboot

Cylosoft · Jun 30, 2023, 1:29 AM

So I have IPv6 disabled. NTP set to use time.google.com. After a reboot NTP tries to use and IPv6 IP and fails. If I restart the NTP service it uses IPv4 and all is well. 23.x versions and 2.7 seem to all do it.

johnpoz · Jun 30, 2023, 1:33 AM

@Cylosoft what does IPv6 disabled mean - if you don't have IPv6 setup, not sure how it would even try Ipv6?

Do you have your wan interface for Ipv6 set to none? Or you have IPv6 disabled via the check box and your still getting a IPv6 on your wan?

What do you have this set too? In the NTP settings

Is your problem with the bootstrap ntp sync at startup?

https://docs.netgate.com/pfsense/en/latest/services/ntpd/bootstrap.html#clock-bootstrap-behavior

Cylosoft · Jun 30, 2023, 3:01 AM

@johnpoz DNS Resolution was set to Auto. So I think you are going to have fixed it for me. Thanks.

In advanced networking I have "allow IPv6" unchecked. Also "Prefer IPv4 over IPv6" checked. But NTP was pulling IPv6 IPs which definitely never work.

johnpoz · Jun 30, 2023, 3:44 AM

@Cylosoft said in NTP with time.google.com after a reboot:

allow IPv6" unchecked.

But was your wan actually getting an IPv6 address?

What do you have set for your wan interface?

If you have no IPv6 address, ntp should never try and talk to a Ipv6 - cuz yeah never going to work ;)

Cylosoft · Jun 30, 2023, 11:47 AM

@johnpoz WAN IPv6 is set to none. I agree there is no way NTP could ever use IPv6. I think the auto setting is broken. If I restart the NTP service it works fine. But coming out of a reboot it always pulls IPv6 and stays unreach/pending until I restart NTP server. I tried this on a 2.7 and 23.x box.

johnpoz · Jun 30, 2023, 11:59 AM

@Cylosoft You could stop resolving AAAA that might prevent it..

Simple

private-address: ::/0

in your unbound config and AAAA will never get resolved, so did setting it to v4 fix it.. That was added a while ago to fix this specific issue I believe.

https://redmine.pfsense.org/issues/10322

I just point pfsense to my local ntp server running on a pi, so I have never seen this issue.

If you used the default ntp pool address, this would never happen either because there is no IPv6 in it, ntp pools only have IPv6 in specific pool address 2 I believe. I don't think any of the pfsense.ntp.pool.org will return Ipv6..

;; QUESTION SECTION:
;2.pool.ntp.org.                        IN      AAAA

;; ANSWER SECTION:
2.pool.ntp.org.         3600    IN      AAAA    2620:2d:4000:1::3f
2.pool.ntp.org.         3600    IN      AAAA    2620:149:a33:3000::1e2
2.pool.ntp.org.         3600    IN      AAAA    2001:470:8d7e:400:20c:29ff:fe86:4251
2.pool.ntp.org.         3600    IN      AAAA    2607:7c80:54:3::32

$ dig 0.pool.ntp.org AAAA

; <<>> DiG 9.16.41 <<>> 0.pool.ntp.org AAAA
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30482
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;0.pool.ntp.org.                        IN      AAAA

Cylosoft · Jun 30, 2023, 11:55 AM

@johnpoz Yes IPv4 setting on NTP fixed everything.

johnpoz · Jun 30, 2023, 12:01 PM

@Cylosoft great to hear, but yeah 2.pfsense.pool.ntp.org does return AAAA

;; QUESTION SECTION:
;2.pfsense.pool.ntp.org.                IN      AAAA

;; ANSWER SECTION:
2.pfsense.pool.ntp.org. 3600    IN      AAAA    2600:3c01::f03c:92ff:fe8c:f7c9
2.pfsense.pool.ntp.org. 3600    IN      AAAA    2620:149:a32:3000::1f2
2.pfsense.pool.ntp.org. 3600    IN      AAAA    2607:f1c0:1801:61::1
2.pfsense.pool.ntp.org. 3600    IN      AAAA    2606:4700:f1::1

but 0, 1 or 3 do not.

Cylosoft · Jun 30, 2023, 12:09 PM

@johnpoz We typically put time.google.com and time.cloudflare.com. In our firewall life previous to PF we had issues with some ntp.org pools and had to switch off. Even though it makes no sense we just still don't use them.

johnpoz · Jun 30, 2023, 12:21 PM

@Cylosoft if you have no IPv6 - you might want to use the unbound thing so no AAAA ever get resolved.. Glad the v4 ntp setting worked for you.

JonathanLee · Jun 30, 2023, 3:30 PM

@johnpoz thanks I wanted to share the results after with a dns lookup or dig with Google

private-address: ::/0
Added to DNS resolver advanced config.

Mission-Ghost · Jul 7, 2023, 3:57 PM

Re: NTP with time.google.com after a reboot

[Sorry for the cross post; I mistakenly used reply to topic but this forum software won’t let me delete my own erroneous post.]

This happened to me as well. I have ipv6 disabled. NTP configured with auto dns was fine until a reboot. It then started trying to open an ipv6 socket on my Starlink interface and failing every minute or so.

I use pool.ntp.org. I’m running pfSense v23.05.1-RELEASE. I also have a T-mobile interface in load balancing configuration.

I’ve experienced this in earlier versions as well.

I set up ipv6 on the starlink interface and ntp stopped generating the error. After finding the above I also set ntp dns to ipv4. I may someday go back and disable IPv6 on the Starlink interface but it doesn’t seem to matter much either way.

It appears ntp auto function has a bug where in some circumstances it attempts to open an IPv6 socket one some external interfaces even when they don’t have IPv6 configured and IPv6 is disabled system wide.

This bug doesn’t seem to interfere with ntp working properly but it does generate thousands of errors. In the interest of making pfSense as bug free as possible this should be investigated and fixed.

JonathanLee · Jul 7, 2023, 4:07 PM

@Mission-Ghost have you tried to use a NAT port forwarding rule? This will redirect all requests for NTP to the server of your choice?

This is what I do as I had many devices requesting NTP all over.

Lan interface -> anything source address inside the Lan net -> over UDP-> any source port -> redirect anything not going to the firewall or loopbacks (any other ntp requests) -> that wants NTP as its destination port

This will redirect any NTP requests that are not being sent to the firewall right back to it. Port forwarding.

(alias has the firewalls address and its loopback, you would need to make 2 one for IPv6 and one rule for IPv4)

Mission-Ghost · Jul 7, 2023, 5:47 PM

@JonathanLee thanks for your post.

Yes, my NTP rules are configured to redirect all of my LAN NTP requests to the router's NTP server. Our LAN devices keep excellent time using this.

However, it doesn't seem to help with or be a factor in the reboot-then-IPv6-socket request error after reboot since NTP configuration.

johnpoz · Jul 7, 2023, 6:53 PM

@Mission-Ghost if it happens on boot - you might want to take a look here

https://docs.netgate.com/pfsense/en/latest/services/ntpd/bootstrap.html

I changed it on mine just recently so it points to my local ntp server vs google time

Mission-Ghost · Jul 7, 2023, 8:08 PM

@johnpoz I wrote up a detailed reply with my troubleshooting and results following your suggestion, but I can't post it. I get a message that the post contains spam courtesy of some (defective) spam filter service. How do I work around this?

Mission-Ghost · Jul 7, 2023, 9:39 PM

@johnpoz thanks for the idea.

I set the conf file with ip addresses of the ntp.org servers plus one time.gov server, disabled the Starlink IPv6 configuration and left the NTP server DNS setting to IPv4 only and rebooted.

The error returned and repeats as before. It looks like this:

Jul 7 13:43:12 	ntpd 	46651 	failed to init interface for address fe80::f2ad:4eff:fe1d:bd1f%12
Jul 7 13:43:12 	ntpd 	46651 	unable to create socket on mvneta0.4092 (31) for fe80::f2ad:4eff:fe1d:bd1f%12#123
...
Jul 7 13:22:51 	ntpd 	46651 	0.0.0.0 c615 05 clock_sync
Jul 7 13:22:51 	ntpd 	46651 	65.100.46.166 101a 8a sys_peer
Jul 7 13:22:49 	ntpd 	46651 	162.159.200.1 0014 84 reachable 
.
.
.
[Bootup and initial NTP sequence...]

So, I've now gone back and re-configured the Starlink interface in accordance with Step 1 in the following procedure as of about 1:52pm:

https://starlinkmag.com/starlink-support-ipv6/

IPv6 remains disabled in the rest of the router's configuration.

1:52pm was about when the next IPv6 failure was due to show up in the log.

It may be meaningful that the retry is exactly every five minutes, like clockwork, to the second.

Returning to the NTP log, it is now 1:57pm and still no more IPv6 error:

Jul 7 13:57:30 	ntpd 	46651 	65.100.46.166 143a 8a sys_peer
Jul 7 13:48:12 	ntpd 	46651 	failed to init interface for address fe80::f2ad:4eff:fe1d:bd1f%12
Jul 7 13:48:12 	ntpd 	46651 	unable to create socket on mvneta0.4092 (32) for fe80::f2ad:4eff:fe1d:bd1f%12#123
...
Jul 7 13:22:51 	ntpd 	46651 	0.0.0.0 c615 05 clock_sync
Jul 7 13:22:51 	ntpd 	46651 	65.100.46.166 101a 8a sys_peer 
.
.
.

So, in my configuration (on a Netgate 1100), I'm concluding so far:

The error only points to the Starlink WAN interface. No error shows for the T-Mobile WAN interface.
Changing the NTP server DNS resolution to IPv4 only fails to eliminate the error.
Creating a /conf/ntp-boot-time-servers file with IPv4 address pointers to various ntp.org and time.gov servers fails to eliminate the error.
Enabling IPv6 on the Starlink interface eliminates the error.

Mission-Ghost · Jul 7, 2023, 9:41 PM

@Mission-Ghost Looks like one can't post more than a few lines of pfSense logs before the forum spam filter misinterprets it as something spammy. So I clipped the log output and it posted without interference from the spam filter.

JonathanLee · Jul 8, 2023, 7:23 AM

@johnpoz I was just going to quote that as I just leaned this came back to this post but you beat me to it

"The firewall performs a one-time sync to multiple NTP servers with static IP addresses from Google Public NTP. This avoids a chicken-and-egg problem where the firewall cannot resolve NTP servers because DNSSEC, which is enabled by default, cannot function when the clock is inaccurate. This happens much later in the boot process because it cannot be performed until after the firewall has configured its interfaces and routing.

This gets the clock as close to accurate as possible without a persistent NTP daemon. There is a hard timeout of 30 seconds in case the upstream servers are unreachable.

Changing Clock Bootstrap Behavior
The NTP clock bootstrap behavior can easily be disabled or changed if an administrator does not want a firewall to contact the default list of servers.

To disable the bootstrap, create the file /conf/ntp-boot-time-servers as an empty file. If the file exists and is empty the firewall will skip the initial sync.

To use alternate servers, create the file /conf/ntp-boot-time-servers and add in one or more IP addresses separated by a single space each. If this file contains a list of space-separated IP addresses, the firewall will use those for the bootstrap sync instead" (Netgate docs).

Mission-Ghost · Jul 8, 2023, 9:33 PM

@JonathanLee I tried creating the file and pointing it to ntp.org and time.gov NTP server IP addresses, but to no effect at preventing the attempts to use a non-existent IPv6 WAN interface address. I'll try an empty file and see if that has an effect and report back.

Even so, I think an issue exists where the NTP server fails to respect the disablement of IPv6 and then infinitely loops trying an IPv6 WAN interface address every five minutes, even if there isn't one.