NTP Issue

admacdo

I'm still running 2.7.2 as everything else seems to be just fine, but things have changed on my network a while ago with yet another stratum 1 NTP server.
I expect I won't be totally happy until I have four, but that will be risking death from my girlfriend, if she ever finds out.
The number of NTP based clocks has also increased from one to four.
I can recommend the "When" iottimer wifi clock from Aliexpress actually. Just follow the instructions about refreshing your browser window once you try changing it to English, or it won't appear to change.

Anyway, the second NTP server has two interfaces which can be independently configured, so today I decided to fire up another firewall interface and allocate it a class B address and directly connect it to the second interface on the new NTP server.

So now I have three local stratum 1 NTP sources configured. All NTP servers show the connections from the firewall on all three interfaces when it polls them every minute.

As far as the firewall is concerned, they're pending/unreachable.
For extra fun, the interface connected solely to the NTP second interface has an explicit permit any NTP rule.

I even tried setting the second NTP server interface to broadcast mode. Zilch.

It's disappointing as prior to 2.7.2, my firewall was sub millisecond synchronised.

stephenw10

Do you see replies back to the fire wall in the states? Or in a pcap?

admacdo

@stephenw10
Nope.
Only the external NTP servers are seen and used.

admacdo

@stephenw10
Might have helped to post this:

09:22:49.170320 60:be:b4:07:c6:17 > a6:4c:5e:80:2e:fb, ethertype IPv4 (0x0800), length 90: (tos 0xb8, ttl 64, id 28353, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.1.123 > 172.16.1.2.123: [bad udp cksum 0x5a6d -> 0x8f85!] NTPv4, Client, length 48
Leap indicator: (0), Stratum 2 (secondary reference), poll 6 (64s), precision -23
Root Delay: 0.015640, Root dispersion: 0.006912, Reference-ID: 0x11fd4225
Reference Timestamp: 3930801750.204506135 (2024-07-24T09:22:30Z)
Originator Timestamp: 3930801704.158000500 (2024-07-24T09:21:44Z)
Receive Timestamp: 3930801704.161610624 (2024-07-24T09:21:44Z)
Transmit Timestamp: 3930801769.170296701 (2024-07-24T09:22:49Z)
Originator - Receive Timestamp: +0.003610124
Originator - Transmit Timestamp: +65.012296201
09:22:49.170381 a6:4c:5e:80:2e:fb > 60:be:b4:07:c6:17, ethertype IPv4 (0x0800), length 90: (tos 0x0, ttl 255, id 10031, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.2.123 > 172.16.1.1.123: [udp sum ok] NTPv4, Server, length 48
Leap indicator: (0), Stratum 1 (primary reference), poll 0 (1s), precision -18
Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: PPS^@
Reference Timestamp: 3930801769.000000000 (2024-07-24T09:22:49Z)
Originator Timestamp: 3930801769.170296701 (2024-07-24T09:22:49Z)
Receive Timestamp: 3930801769.167000500 (2024-07-24T09:22:49Z)
Transmit Timestamp: 3930801769.167000500 (2024-07-24T09:22:49Z)
Originator - Receive Timestamp: -0.003296201
Originator - Transmit Timestamp: -0.003296201
09:22:57.187023 60:be:b4:07:c6:17 > a6:4c:5e:80:2e:fb, ethertype IPv4 (0x0800), length 90: (tos 0xb8, ttl 64, id 45902, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.1.123 > 172.16.1.2.123: [bad udp cksum 0x5a6d -> 0x5082!] NTPv4, Client, length 48
Leap indicator: (0), Stratum 2 (secondary reference), poll 6 (64s), precision -23
Root Delay: 0.015640, Root dispersion: 0.007034, Reference-ID: 0x11fd4225
Reference Timestamp: 3930801750.204506135 (2024-07-24T09:22:30Z)
Originator Timestamp: 3930801769.167000500 (2024-07-24T09:22:49Z)
Receive Timestamp: 3930801769.170386584 (2024-07-24T09:22:49Z)
Transmit Timestamp: 3930801777.187008800 (2024-07-24T09:22:57Z)
Originator - Receive Timestamp: +0.003386084
Originator - Transmit Timestamp: +8.020008299
09:22:57.187083 a6:4c:5e:80:2e:fb > 60:be:b4:07:c6:17, ethertype IPv4 (0x0800), length 90: (tos 0x0, ttl 255, id 10032, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.2.123 > 172.16.1.1.123: [udp sum ok] NTPv4, Server, length 48
Leap indicator: (0), Stratum 1 (primary reference), poll 0 (1s), precision -18
Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: PPS^@
Reference Timestamp: 3930801777.000000000 (2024-07-24T09:22:57Z)
Originator Timestamp: 3930801777.187008800 (2024-07-24T09:22:57Z)
Receive Timestamp: 3930801777.183000500 (2024-07-24T09:22:57Z)
Transmit Timestamp: 3930801777.183000500 (2024-07-24T09:22:57Z)
Originator - Receive Timestamp: -0.004008300
Originator - Transmit Timestamp: -0.004008300
09:23:05.179910 60:be:b4:07:c6:17 > a6:4c:5e:80:2e:fb, ethertype IPv4 (0x0800), length 90: (tos 0xb8, ttl 64, id 43251, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.1.123 > 172.16.1.2.123: [bad udp cksum 0x5a6d -> 0x5129!] NTPv4, Client, length 48
Leap indicator: (0), Stratum 2 (secondary reference), poll 6 (64s), precision -23
Root Delay: 0.015640, Root dispersion: 0.007156, Reference-ID: 0x11fd4225
Reference Timestamp: 3930801750.204506135 (2024-07-24T09:22:30Z)
Originator Timestamp: 3930801777.183000500 (2024-07-24T09:22:57Z)
Receive Timestamp: 3930801777.187088760 (2024-07-24T09:22:57Z)
Transmit Timestamp: 3930801785.179879918 (2024-07-24T09:23:05Z)
Originator - Receive Timestamp: +0.004088260
Originator - Transmit Timestamp: +7.996879418
09:23:05.179988 a6:4c:5e:80:2e:fb > 60:be:b4:07:c6:17, ethertype IPv4 (0x0800), length 90: (tos 0x0, ttl 255, id 10033, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.2.123 > 172.16.1.1.123: [udp sum ok] NTPv4, Server, length 48
Leap indicator: (0), Stratum 1 (primary reference), poll 0 (1s), precision -18
Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: PPS^@
Reference Timestamp: 3930801785.000000000 (2024-07-24T09:23:05Z)
Originator Timestamp: 3930801785.179879918 (2024-07-24T09:23:05Z)
Receive Timestamp: 3930801785.176000500 (2024-07-24T09:23:05Z)
Transmit Timestamp: 3930801785.176000500 (2024-07-24T09:23:05Z)
Originator - Receive Timestamp: -0.003879418
Originator - Transmit Timestamp: -0.003879418
09:24:11.223769 60:be:b4:07:c6:17 > a6:4c:5e:80:2e:fb, ethertype IPv4 (0x0800), length 90: (tos 0xb8, ttl 64, id 17146, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.1.123 > 172.16.1.2.123: [bad udp cksum 0x5a6d -> 0x2cfc!] NTPv4, Client, length 48
Leap indicator: (0), Stratum 2 (secondary reference), poll 6 (64s), precision -23
Root Delay: 0.015640, Root dispersion: 0.008148, Reference-ID: 0x11fd4225
Reference Timestamp: 3930801750.204506135 (2024-07-24T09:22:30Z)
Originator Timestamp: 3930801785.176000500 (2024-07-24T09:23:05Z)
Receive Timestamp: 3930801785.180030896 (2024-07-24T09:23:05Z)
Transmit Timestamp: 3930801851.223755123 (2024-07-24T09:24:11Z)
Originator - Receive Timestamp: +0.004030396
Originator - Transmit Timestamp: +66.047754622
09:24:11.223843 a6:4c:5e:80:2e:fb > 60:be:b4:07:c6:17, ethertype IPv4 (0x0800), length 90: (tos 0x0, ttl 255, id 10034, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.2.123 > 172.16.1.1.123: [udp sum ok] NTPv4, Server, length 48
Leap indicator: (0), Stratum 1 (primary reference), poll 0 (1s), precision -18
Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: PPS^@
Reference Timestamp: 3930801851.000000000 (2024-07-24T09:24:11Z)
Originator Timestamp: 3930801851.223755123 (2024-07-24T09:24:11Z)
Receive Timestamp: 3930801851.220000500 (2024-07-24T09:24:11Z)
Transmit Timestamp: 3930801851.220000500 (2024-07-24T09:24:11Z)
Originator - Receive Timestamp: -0.003754622
Originator - Transmit Timestamp: -0.003754622
09:24:19.195033 60:be:b4:07:c6:17 > a6:4c:5e:80:2e:fb, ethertype IPv4 (0x0800), length 90: (tos 0xb8, ttl 64, id 50821, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.1.123 > 172.16.1.2.123: [bad udp cksum 0x5a6d -> 0xae46!] NTPv4, Client, length 48
Leap indicator: (0), Stratum 2 (secondary reference), poll 6 (64s), precision -23
Root Delay: 0.015640, Root dispersion: 0.008270, Reference-ID: 0x11fd4225
Reference Timestamp: 3930801750.204506135 (2024-07-24T09:22:30Z)
Originator Timestamp: 3930801851.220000500 (2024-07-24T09:24:11Z)
Receive Timestamp: 3930801851.223882897 (2024-07-24T09:24:11Z)
Transmit Timestamp: 3930801859.195007028 (2024-07-24T09:24:19Z)
Originator - Receive Timestamp: +0.003882397
Originator - Transmit Timestamp: +7.975006527
09:24:19.195121 a6:4c:5e:80:2e:fb > 60:be:b4:07:c6:17, ethertype IPv4 (0x0800), length 90: (tos 0x0, ttl 255, id 10035, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.2.123 > 172.16.1.1.123: [udp sum ok] NTPv4, Server, length 48
Leap indicator: (0), Stratum 1 (primary reference), poll 0 (1s), precision -18
Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: PPS^@
Reference Timestamp: 3930801859.000000000 (2024-07-24T09:24:19Z)
Originator Timestamp: 3930801859.195007028 (2024-07-24T09:24:19Z)
Receive Timestamp: 3930801859.192000500 (2024-07-24T09:24:19Z)
Transmit Timestamp: 3930801859.192000500 (2024-07-24T09:24:19Z)
Originator - Receive Timestamp: -0.003006528
Originator - Transmit Timestamp: -0.003006528
09:24:27.176376 60:be:b4:07:c6:17 > a6:4c:5e:80:2e:fb, ethertype IPv4 (0x0800), length 90: (tos 0xb8, ttl 64, id 48593, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.1.123 > 172.16.1.2.123: [bad udp cksum 0x5a6d -> 0x37e1!] NTPv4, Client, length 48
Leap indicator: (0), Stratum 2 (secondary reference), poll 6 (64s), precision -23
Root Delay: 0.015640, Root dispersion: 0.008392, Reference-ID: 0x11fd4225
Reference Timestamp: 3930801750.204506135 (2024-07-24T09:22:30Z)
Originator Timestamp: 3930801859.192000500 (2024-07-24T09:24:19Z)
Receive Timestamp: 3930801859.195166604 (2024-07-24T09:24:19Z)
Transmit Timestamp: 3930801867.176368362 (2024-07-24T09:24:27Z)
Originator - Receive Timestamp: +0.003166103
Originator - Transmit Timestamp: +7.984367861
09:24:27.176494 a6:4c:5e:80:2e:fb > 60:be:b4:07:c6:17, ethertype IPv4 (0x0800), length 90: (tos 0x0, ttl 255, id 10036, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.2.123 > 172.16.1.1.123: [udp sum ok] NTPv4, Server, length 48
Leap indicator: (0), Stratum 1 (primary reference), poll 0 (1s), precision -18
Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: PPS^@
Reference Timestamp: 3930801867.000000000 (2024-07-24T09:24:27Z)
Originator Timestamp: 3930801867.176368362 (2024-07-24T09:24:27Z)
Receive Timestamp: 3930801867.173000500 (2024-07-24T09:24:27Z)
Transmit Timestamp: 3930801867.173000500 (2024-07-24T09:24:27Z)
Originator - Receive Timestamp: -0.003367861
Originator - Transmit Timestamp: -0.003367861
09:25:34.172849 60:be:b4:07:c6:17 > a6:4c:5e:80:2e:fb, ethertype IPv4 (0x0800), length 90: (tos 0xb8, ttl 64, id 10788, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.1.123 > 172.16.1.2.123: [bad udp cksum 0x5a6d -> 0x34ff!] NTPv4, Client, length 48
Leap indicator: (0), Stratum 2 (secondary reference), poll 6 (64s), precision -23
Root Delay: 0.015640, Root dispersion: 0.009384, Reference-ID: 0x11fd4225
Reference Timestamp: 3930801750.204506135 (2024-07-24T09:22:30Z)
Originator Timestamp: 3930801867.173000500 (2024-07-24T09:24:27Z)
Receive Timestamp: 3930801867.176498219 (2024-07-24T09:24:27Z)
Transmit Timestamp: 3930801934.172838781 (2024-07-24T09:25:34Z)
Originator - Receive Timestamp: +0.003497719
Originator - Transmit Timestamp: +66.999838281
09:25:34.172908 a6:4c:5e:80:2e:fb > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 172.16.1.1 tell 172.16.1.2, length 46
09:25:34.172917 60:be:b4:07:c6:17 > a6:4c:5e:80:2e:fb, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 172.16.1.1 is-at 60:be:b4:07:c6:17, length 28
09:25:34.172956 a6:4c:5e:80:2e:fb > 60:be:b4:07:c6:17, ethertype IPv4 (0x0800), length 90: (tos 0x0, ttl 255, id 10037, offset 0, flags [none], proto UDP (17), length 76)
172.16.1.2.123 > 172.16.1.1.123: [udp sum ok] NTPv4, Server, length 48
Leap indicator: (0), Stratum 1 (primary reference), poll 0 (1s), precision -18
Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: PPS^@
Reference Timestamp: 3930801934.000000000 (2024-07-24T09:25:34Z)
Originator Timestamp: 3930801934.172838781 (2024-07-24T09:25:34Z)
Receive Timestamp: 3930801934.169000500 (2024-07-24T09:25:34Z)
Transmit Timestamp: 3930801934.169000500 (2024-07-24T09:25:34Z)
Originator - Receive Timestamp: -0.003838281
Originator - Transmit Timestamp: -0.003838281

johnpoz

@admacdo said in NTP Issue:

[bad udp cksum 0x5a6d

odd that all the traffic you send to 1.2 shows bad checksum??

But traffic from 1.2 to 1.1 shows ok?

Do you have your checksum offload on?

admacdo

@johnpoz
No checksum offload tick.

stephenw10

@admacdo said in NTP Issue:

No checksum offload tick.

'No tick' could imply it's not disabled and therefore is offloaded.

admacdo

@stephenw10
Yeah, bad explanation on my part. It's not disabled.
I even cycled it on and off a few times with a reboot to see if it made any difference.

No change.

I'm a little disappointed.

stephenw10

If it's actually disabled you can see that directly using ifconfig at the CLI, for example:

[24.08-DEVELOPMENT][admin@4200.stevew.lan]/root: ifconfig -m igc1
igc1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	options=48020b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,HWSTATS,MEXTPG>
	capabilities=4f43fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>

Where tx and rx checksums are shown in capabilities but not options. Though you see vlan checkums are still enabled.

admacdo

@stephenw10

Well, it looks like it's not disabled, as all previous indications have said that it isn't.

igc2: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: LAN
options=4e020bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
capabilities=4f43fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 60:be:b4:07:c6:16
inet 192.168.25.254 netmask 0xffffff00 broadcast 192.168.25.255
inet6 fe80::62be:b4ff:fe07:c616%igc2 prefixlen 64 scopeid 0x3
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
supported media:
media autoselect
media 2500Base-T
media 1000baseT
media 1000baseT mediaopt full-duplex
media 100baseTX mediaopt full-duplex
media 100baseTX
media 10baseT/UTP mediaopt full-duplex
media 10baseT/UTP
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

stephenw10

Ok then the checksum failure in a pcap is expected. That isn't really a problem normally though.

admacdo

Well, just for more fun, a couple of weeks ago, I bought an SG-1100 as I needed a backup firewall in case mine self toasts somehow.
I reset it to defaults, stuck a rough config on and as a quick test, pointed it at both my local NTP stratum one time servers and also time.apple.com.

The apple timeserver responded instantly and the 1100 totally ignored the two locals.

I updated it to the latest Pfsense Plus and the same thing happened.

Today I was given a VK-T40E2 by a colleague who'd upgraded his home firewall setup.
I reset it to defaults and configured it, then pointed it at the same three timeservers as the 1100.
It instantly latched onto one of the locals while still seeing the other two.
It was still on the original version, so I updated that to the latest legacy 2.3.5 (I think). Going further will require a complete reinstall and I don't have a null modem cable to hand these days.
It still saw all the local and internet NTP servers that I cared to list.

So of course, I can't wait to get it to 2.4 so that it can't see the two local stratum ones...
At least that's what I'm betting will happen.

Gertjan

@admacdo said in NTP Issue:

The apple timeserver responded instantly and the 1100 totally ignored the two locals.

It will put the other two as a 'hot spare'.
Check for yourself (example) :

Btw : look at that : NTP went IPv6 only also ...

@admacdo said in NTP Issue:

I updated it to the latest Pfsense Plus

24.11.

@admacdo said in NTP Issue:

the same three timeservers

Normally, you shouldn't use 3 'off something' (DNS, NTP, whatever) as it created problems.
See it like this : you have to buy a new car, but you want to think about it, so your interrogations @home and discuss it with your wive.
What would happen if you had 3 wives ?
I can tell you what will happen. You won't drive any car any soon (you couldn't afford one anyway ^^)

So :
NTP : pick a 'pool'.
DNS : stick to 'root servers' (a pool actually).
etc etc.

@admacdo said in NTP Issue:

latest legacy 2.3.5

An APU ? 32 bits ??

@admacdo said in NTP Issue:

it to 2.4 so

2.4 ?

admacdo

@Gertjan

Go and read the entire thread, start to finish. Pay particular attention to what I have already posted.
Go and read how NTP actually works.
Not SNTP.
The REAL complete NTP and learn EXACTLY WHY you put in multiple NTP sources.
Then maybe go and read why Industry Best Practice requires four.

admacdo

@ARAMP1

And sadly, the issue still exists with 2.8

There's just no way that I can find to get pfsense back to taking notice of any NTP servers on the LAN.

stephenw10

Ok reading back a few things are unclear here.

The configures internal ntp server(s) just show as unreaching in pfSense?

A pcap on the ntp server shows requests from pfSense and responses from the server?

But the state table on the firewall only shows one way traffic?

Does a pcap on pfSense show the replies from server?

Does it do the same thing if there are no other servers configured?

stephenw10

It still works fine for me here from 2.8 and 25.03:

[2.8.0-RELEASE][admin@cedev-2.stevew.lan]/root: ntpq -np
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*172.21.16.1     .GPS.            1 u   43   64  377    0.378   -0.347   2.602

Now the jitter is crap on that server because it's a USB connected GPS. My real GPS device failed a while back

So in a list it doesn't get selected as the active peer. But it still polls it just fine:

[2.8.0-RELEASE][admin@t70.stevew.lan]/root: ntpq -np
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
-172.21.16.1     .GPS.            1 u   43   64  377    0.404   -2.999   0.919
 2.pfsense.pool. .POOL.          16 p    -   64    0    0.000   +0.000   0.000
+217.154.60.177  139.143.5.31     2 u   26   64  377    8.644   +1.450   0.256
*85.199.214.99   .GPS.            1 u   22   64  377    6.458   +1.507   0.204
+185.83.169.27   .GPS.            1 u   22   64  377   10.610   +1.119   0.366

That also includes whether or not the server is WAN or LAN side, as long as the ACLs allow it.

admacdo

@stephenw10

A brief recap of the situation.
I run two LAN based GPS ntp servers.
They used to run fine before I updated to 2.7.
They would instantly get selected as peer and next best within a very short time.
Other ntp servers on the internet were also selected so as to show internet conditions within the state, country and various parts of the world.
As soon as the 2.7 update happened, the LAN based NTP servers became unreachable.
There have been at least three other people in the world who had this exact same issue.

The NTP servers each remain available to each of the 72 devices currently on the home network.
My particular box does not like USB GPS devices and the serial port is an RJ45 port, which never responded to any combination of mapping the PPS, so that option is out of the window.

So with your real GPS device failed, how can you be sure that your install still works with a LAN based NTP server?

stephenw10

That is an LAN based ntp server I've shown above. It's a crap ntp server but it's still local and is reachable just fine.

So something about your servers is different, hence my list of questions. When I read back through the thread at one point you said the state table showed one way traffic. That implies some network issue rather than anything ntp specific.

I'll try and repair my true GPS module. That used pps over a real serial port. Unfortunately the antenna connector was pulled off the board so it requires some delicate surgery!

admacdo

@stephenw10
Well, I'm amazed that you have a LAN based NTP server turning up as being available.

I can see that my pfsense box is contacting both my ntp servers because they report the last 100 hosts.
I have opt 2 configured with zero firewall rules and it goes to the second port of the main NTP server. That port serves only port 123, so it's a handy test.
That port of the NTP server is configured to broadcast NTP every thirty seconds AND I can see that traffic turn up as measured by the firewall, every 30 seconds. I ditched the broadcast and seeing as nothing but NTP traffic is on that port, it clearly shows traffic going to and from the NTP server.
So we know that the the network is just fine between the firewall and three ports between both NTP servers.

A little while ago, I disabled all the LAN firewall rules, then restarted the NTP service.

As per usual, all of the LAN NTP servers were unreachable and yet time.apple.com quickly was selected out of the six other options.

(For anyone who cares, I logged various NTP servers around the world for over two and half years at my work. time.apple.com is by far the most reliable and fastest. I'm pretty sure they have a decent time server in most countries, even if the traceroute takes you back to the USA. Not ONE of ANY of the public pools of NTP servers were reliable. I would NEVER recommend using pools of any NTP servers.

I've established several times to my satisfaction that my firewall is contacting both NTP servers and totally ignoring them.
I'm thinking that it's a kernel level problem now.