StarLink as source for NTP

chpalmer

@stephenw10 said in StarLink as source for NTP:

One of our devs has starlink and has replicated the behavior reported here. We are digging into it.

Any updates to this?

stephenw10

Nothing yet. It was replicated but the cause was not obvious. Once this point release out we should be able to revisit and dig deeper.

Ramosel

@stephenw10 said in StarLink as source for NTP:

Nothing yet. It was replicated but the cause was not obvious. Once this point release out we should be able to revisit and dig deeper.

Since it's not "in the works" currently, I was curious as to whether the Network Time Foundation was even aware of this issue/limitation with NTPd? So I sent a quick 1 page whitepaper to them to introduce it. Not as a complaint, just and introduction.

Maybe that is where the resolution will find a home.

Again, thanks for your help Stephen!
I hope Netgate appreciates you as much as we do!

Rick

davehart

@Ramosel I'm one of the ntpd developers. My hunch is you hone in on what's going awry by stopping the ntpd daemon and then run it from the command line with something like:

sudo sh -c 'ntpd -ngqc /dev/null -D3 192.168.100.1 | tee -a ntpdebug.log 2>&1'

And then kill it after it rejects at least one response from the dish. I gather pfSense is using ntpd 4.2.8p18, and my hunch is this is a regression from earlier ntpd. I recall there was a questionable change to be picky about accepting responses that differ from what the dev who made the change thought was correct. It might be to do with one of the timestamps.

I'd be happy to take a look if someone shares that voluminous trace somewhere I can get it, privately to davehart@gmail.com if they'd rather not dig through it to see if they want to redact anything and are willing to take a chance on my discretion...

Cheers,
Dave Hart

davehart

@Mission-Ghost said in StarLink as source for NTP:

     192.168.100.1.123 > 100.117.x.x.123: [udp sum ok] NTPv4, Server, length 48
 	Leap indicator:  (0), Stratum 1 (primary reference), poll 0 (1s), precision -9

The poll of 0 (2^0 or 1s) is the problem, a regression introduced in 4.2.8p16. See https://bugs.ntp.org/3870.

Ramosel

@davehart said in StarLink as source for NTP:

@Ramosel I'm one of the ntpd developers. My hunch is you hone in on what's going awry by stopping the ntpd daemon and then run it from the command line with something like:
sudo sh -c 'ntpd -ngqc /dev/null -D3 192.168.100.1 | tee -a ntpdebug.log 2>&1'
And then kill it after it rejects at least one response from the dish. I gather pfSense is using ntpd 4.2.8p18, and my hunch is this is a regression from earlier ntpd. I recall there was a questionable change to be picky about accepting responses that differ from what the dev who made the change thought was correct. It might be to do with one of the timestamps.

I'd be happy to take a look if someone shares that voluminous trace somewhere I can get it, privately to davehart@gmail.com if they'd rather not dig through it to see if they want to redact anything and are willing to take a chance on my discretion...

Cheers,
Dave Hart

Wow, thanks Dave!
I'm glad someone from NTF read my submission and looked at this!! I responded to Dru.
I'm not ashamed to admit this request is a bit (not totally) over my head.
Stephen, could you or one of the other code slingers with StarLink on this thread get Dave the information he is asking for? If not, I'll be glad to work with someone one on one to make it happen. Let me know.

Rick

davehart

said in StarLink as source for NTP:

@Mission-Ghost said in StarLink as source for NTP:
     192.168.100.1.123 > 100.117.x.x.123: [udp sum ok] NTPv4, Server, length 48
 	Leap indicator:  (0), Stratum 1 (primary reference), poll 0 (1s), precision -9
The poll of 0 (2^0 or 1s) is the problem, a regression introduced in 4.2.8p16. See https://bugs.ntp.org/3870.

Even better, see https://bugs.ntp.org/3877. There's a patch attached which should resolve the problem. It's been ready to be pulled by the creator of this mess since June of 2024.

Mission-Ghost

@davehart do you still need the debug log? I’ll produce it later today if so.

stephenw10

It looks pretty clear that that is the cause here. But more logs can't hurt! Should confirm it I'd expect.

davehart

@Mission-Ghost said in StarLink as source for NTP:

@davehart do you still need the debug log? I’ll produce it later today if so.

No need, I wouldn't spend time on it. I would like to confirm that it's ntpd 4.2.8p18, and get the output of uname -a

I can crank out a fixed x64 binary with that info and post it.

stephenw10

In CE 2.8.1:

[2.8.1-RELEASE][admin@t70.stevew.lan]/root: pkg info ntp
ntp-4.2.8p18_5
Name           : ntp
Version        : 4.2.8p18_5
Installed on   : Thu May 22 14:10:43 2025 BST
Origin         : net/ntp
Architecture   : FreeBSD:15:amd64
Prefix         : /usr/local
Categories     : net
Licenses       : 
Maintainer     : cy@FreeBSD.org
WWW            : https://www.ntp.org/
Comment        : The Network Time Protocol Distribution
Options        :
	ACTS           : on
	ARBITER        : on
	ARCRON_MSF     : on
	AS2201         : on
	ATOM           : on
	AUDIO_CHU      : off
	BANCOMM        : off
	CHRONOLOG      : on
	CHU            : on
	COMPUTIME      : off
	DATUM          : on
	DCF7000        : off
	DEBUG          : off
	DOCS           : off
	DUMBCLOCK      : on
	EXAMPLES       : off
	FG             : on
	GPSVME         : off
	HEATH          : on
	HOPF6021       : off
	HOPFPCI        : on
	HOPFSERIAL     : on
	HPGPS          : on
	IPV6           : on
	IRIG           : on
	JJY            : on
	JUPITER        : on
	LEITCH         : on
	LOCAL_CLOCK    : on
	MD5            : off
	MEINBERG       : off
	MX4200         : off
	NEOCLOCK4X     : on
	NLS            : on
	NMEA           : on
	NTPSNMPD       : off
	NTP_SIGND      : off
	ONCORE         : on
	PALISADE       : on
	PCF            : on
	PERL_UTILS     : off
	PST            : on
	RAWDCF         : off
	RCC8000        : off
	RIPENCC        : off
	SCHMID         : off
	SEL240X        : off
	SHM            : on
	SPECTRACOM     : on
	SSL            : on
	THREADS        : on
	TRIMTAIP       : off
	TRIMTSIP       : off
	TRUETIME       : on
	ULINK          : on
	VARITEXT       : off
	WHARTON        : off
	WWV            : on
	ZYFER          : on
Shared Libs required:
	libintl.so.8
	libevent_pthreads-2.1.so.7
	libevent_core-2.1.so.7
	libedit.so.0
Annotations    :
	FreeBSD_version: 1500029
	build_timestamp: 2025-05-22T00:17:18+00:00
	built_by       : poudriere-git-3.4.99.20250209
	cpe            : cpe:2.3:a:ntp:ntp:4.2.8p18:::::freebsd15:x64:5
	port_checkout_unclean: no
	port_git_hash  : bc3cd471ba72
	ports_top_checkout_unclean: no
	ports_top_git_hash: 87aee0bc23aa
	repo_type      : binary
	repository     : pfSense
Flat size      : 1.71MiB
Description    :
The Network Time Protocol (NTP) is used to synchronize the time of a
computer client or server to another server or reference time source,
such as a radio or satellite receiver or modem.

It provides client accuracies typically within a millisecond on LANs
and up to a few tens of milliseconds on WANs relative to a primary
server synchronized to Coordinated Universal Time (UTC) via a Global
Positioning Service (GPS) receiver, for example.

Typical NTP configurations utilize multiple redundant servers and diverse
network paths, in order to achieve high accuracy and reliability.

Some configurations include cryptographic authentication to prevent
accidental or malicious protocol attacks.

[2.8.1-RELEASE][admin@t70.stevew.lan]/root: uname -a
FreeBSD t70.stevew.lan 15.0-CURRENT FreeBSD 15.0-CURRENT #21 RELENG_2_8_1-n256095-47c932dcc0e9: Thu Aug 28 16:27:48 UTC 2025     root@pfsense-build-release-amd64-1.eng.atx.netgate.com:/var/jenkins/workspace/pfSense-CE-snapshots-2_8_1-main/obj/amd64/AupY3aTL/var/jenkins/workspace/pfSense-CE-snapshots-2_8_1-main/sources/FreeBSD-src-RELENG_2_8_1/amd64.amd64/sys/pfSense amd64

stephenw10

We are looking at building a package with that patch. That will probably be for 26.03-dev though.

cmcdonald

I have Starlink:

Before patch:

After patch:

Another question here for @davehart , should I advocate for including this patch in the upstream FreeBSD port (net/ntp) ? I can make that happen if that is the direction we want to take.

stephenw10

Nice! Looks like some tweaks to the data/parsing required too.

davehart

@cmcdonald said in StarLink as source for NTP:

should I advocate for including this patch in the upstream FreeBSD port (net/ntp) ?

Thanks for verifying that resolves the problem. I'm sure there are many others using pfSense who will benefit.

I have no objection to FreeBSD picking up any of the many pending ntp patches while we wait for NTF/ntp.org to get another release out. It's their call if it's worth their time and safe enough.

cmcdonald

@stephenw10 said in StarLink as source for NTP:

Nice! Looks like some tweaks to the data/parsing required too.

/usr/local/sbin/ntpq -pnw
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*192.168.100.1                    1 u   57   64  377    0.546   +0.120   7.203

refid is empty now

tinfoilmatt

You mean it's just not being parsed?

cmcdonald

@tinfoilmatt said in StarLink as source for NTP:

You mean it's just not being parsed?

pfSense expects refid to not be empty, so it ends up shifting all other columns over.

Is there ever a legit condition where refid is indeed empty?

I'm not sure.

tinfoilmatt

Is there ever a legit condition where refid is indeed empty?

No idea. Your comment piqued my curiosity only because this post seems to demonstrate that the StarLink NTP implementation does, in fact, reply with an REFID of ".XFAC."?

Can't read myself since the network I'm on at the moment has the domain www.ntp.org blocked ()—but the search engine description of whatever this page contains regarding REFID seems relevant:

https://www.ntp.org/documentation/ietf/ntp-refid-updates/

davehart

@cmcdonald said in StarLink as source for NTP:

pfSense expects refid to not be empty, so it ends up shifting all other columns over.

Is there ever a legit condition where refid is indeed empty?

ntpd displays text refids bracketed by periods. It seems reasonable to do that for refids with only unprintable/invisible characters. Stratum 1 sources have text refids like .GPS.

Feel free to file a bug report at https://bugs.ntp.org/