23.09 Local NTP server "unrechable" (worked with 23.05.1)

[conover]

After upgrading from 23.05.1 to 23.09 I discovered that my local NTP server (small PPS appliance) has now the status "unreach/pendig". Worked perfectly with < 23.09, did not change anything after upgrading.
The NTP server is still alive and the (very limited) logging on it shows also connections from my pfSense boxes (6100).

Pointing other clients to that NTP server directly (instead using a pfSense box) works also (they are getting the correct time from it).

Any (significant) changes in 23.09 NTPd I missed, so pfSense will not synchronize with local NTP servers anymore? Or any other configuration changes that have an impact on that?

Thanks!

[johnpoz]

@conover I sync to a local ntp, and all seems to be working fine

Maybe you want to sniff the ntp traffic.. So you can see what is answered back, if anything, etc.

Your showing a reach of 0 I take it.

I did not need to make any changes after upgrading to 23.09 from 23.05.1, I do believe that ntp did update to a newer version..

edit: here I did a sniff of my pfsense talking to my local ntp as example

edit: yeah I do believe there was a slight update

[23.09-RELEASE][admin@sg4860.local.lan]/root: ntpd --help
ntpd - NTP daemon program - Ver. 4.2.8p17

p17 came out in june of this year.. Before it might of been p15 I think??? Which reminds me should check if updated my clients on my network.. Just checked my windows box and its still using p15

[conover]

You are right, reach is actually 0

I used pfSense's packet capture, and pfSense (192.168.168.254) will get a reasoanble answer from the NTP server (192.168.168.218).
I am not an NTP expert but looks good to me...

15:13:44.908221 90:ec:77:34:f1:eb > d8:b0:4c:ff:06:7c, ethertype IPv4 (0x0800), length 90: (tos 0xb8, ttl 64, id 56553, offset 0, flags [none], proto UDP (17), length 76)
192.168.168.254.123 > 192.168.168.218.123: [udp sum ok] NTPv4, Client, length 48
Leap indicator: clock unsynchronized (192), Stratum 0 (unspecified), poll 6 (64s), precision -23
Root Delay: 0.000000, Root dispersion: 0.000015, Reference-ID: (unspec)
Reference Timestamp: 0.000000000
Originator Timestamp: 0.000000000
Receive Timestamp: 0.000000000
Transmit Timestamp: 3909222824.908183891 (2023-11-17T15:13:44Z)
Originator - Receive Timestamp: 0.000000000
Originator - Transmit Timestamp: 3909222824.908183891 (2023-11-17T15:13:44Z)
15:13:44.908326 d8:b0:4c:ff:06:7c > 90:ec:77:34:f1:eb, ethertype IPv4 (0x0800), length 90: (tos 0x0, ttl 255, id 14, offset 0, flags [none], proto UDP (17), length 76)
192.168.168.218.123 > 192.168.168.254.123: [udp sum ok] NTPv4, Server, length 48
Leap indicator: (0), Stratum 1 (primary reference), poll 0 (1s), precision -18
Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: PPS^@
Reference Timestamp: 3909222824.000000000 (2023-11-17T15:13:44Z)
Originator Timestamp: 3909222824.908183891 (2023-11-17T15:13:44Z)
Receive Timestamp: 3909222824.903999999 (2023-11-17T15:13:44Z)
Transmit Timestamp: 3909222824.903999999 (2023-11-17T15:13:44Z)
Originator - Receive Timestamp: -0.004183891
Originator - Transmit Timestamp: -0.004183891

[stephenw10]

Nothing I'm aware of. Is pfSense avble to reach that device genarally, ping it?

Do you see responses coming back to pfSense if you pcap for it or check the state?

[conover]

@stephenw10 said in 23.09 Local NTP server "unrechable" (worked with 23.05.1):

Nothing I'm aware of. Is pfSense avble to reach that device genarally, ping it?

Do you see responses coming back to pfSense if you pcap for it or check the state?

Yes it's pingable and pfSense gets back a (reasonable) response, see post above

[johnpoz]

@conover said in 23.09 Local NTP server "unrechable" (worked with 23.05.1):

Leap indicator: clock unsynchronized

Well that is saying the clock is not synchronized, so yeah don't think pfsense would consider that a valid time source to sync too.

edit: oh that was your client.. Doh.. let me look a bit closer.. your getting zero for reach?

[stephenw10]

Sorry left the reply open too long.

[conover]

@johnpoz said in 23.09 Local NTP server "unrechable" (worked with 23.05.1):

@conover said in 23.09 Local NTP server "unrechable" (worked with 23.05.1):

Leap indicator: clock unsynchronized

Well that is saying the clock is not synchronized, so yeah don't think pfsense would consider that a valid time source to sync too.

Thanks, but the "clock unsynchronized" message is part of the packet from pfSense (.254) to the NTP server (.218)?

192.168.168.254.123 > 192.168.168.218.123: [udp sum ok] NTPv4, Client, length 48
Leap indicator: clock unsynchronized (192), Stratum 0 (unspecified), poll 6 (64s), precision -23
    [...]

The answer is

192.168.168.218.123 > 192.168.168.254.123: [udp sum ok] NTPv4, Server, length 48
Leap indicator:  (0), Stratum 1 (primary reference), poll 0 (1s), precision -18
    [...]

Or am I totally wrong?

[johnpoz]

@conover yeah I caught that was the client - see my edit ;)

[conover]

@johnpoz said in 23.09 Local NTP server "unrechable" (worked with 23.05.1):

edit: oh that was your client.. Doh.. let me look a bit closer.. your getting zero for reach?

yes,
"Status / NTP" says Reach 0 for that NTP server (.218),
RefID is ".INIT."

[conover]

Interesting, explicitly using ntpdate works with that NTP server:

[johnpoz]

@conover do you have it set as peer vs server? I can duplicate your issue if set mine to peer vs server

[conover]

@johnpoz Thanks! It was set to server in 23.05 config and it is unchanged. But out of curiosity I also tried "peer" with no change.

[johnpoz]

@conover so you flipped it to peer, and then you flipped it back?

What does your xml show? And your conf file

[23.09-RELEASE][admin@sg4860.local.lan]/etc: cat /var/etc/ntpd.conf 
# 
# pfSense ntp configuration file 
# 

tinker panic 0 

# Orphan mode stratum and Maximum candidate NTP peers
tos orphan 12 maxclock 5


# Upstream Servers
server -4 192.168.3.32 iburst minpoll 6 maxpoll 10 prefer


enable stats
statistics clockstats loopstats peerstats
statsdir /var/log/ntp
logconfig =syncall +clockall +peerall +sysall
driftfile /var/db/ntpd.drift
restrict default kod limited nomodify nopeer notrap
restrict -6 default kod limited nomodify nopeer notrap
interface ignore all
interface ignore wildcard
interface listen igb3
interface listen igb0
interface listen igb4
interface listen igb2
interface listen igb2.6
interface listen igb2.4
interface listen igb5
interface listen lo0
[23.09-RELEASE][admin@sg4860.local.lan]/etc: 

[conover]

@johnpoz said in 23.09 Local NTP server "unrechable" (worked with 23.05.1):

@conover so you flipped it to peer, and then you flipped it back?

Yes, exactly (and restarted the service after each switch to be sure)

What does your xml show? And your conf file

Which XML do you mean?

The conf file looks pretty much the same (but no minpoll) :

# 
# pfSense ntp configuration file 
# 

tinker panic 0 

# Orphan mode stratum and Maximum candidate NTP peers
tos orphan 12 maxclock 5


# Upstream Servers
pool de.pool.ntp.org iburst maxpoll 9
server 192.168.168.218 iburst maxpoll 9 prefer


statsdir /var/log/ntp
logconfig =syncall +clockall
driftfile /var/db/ntpd.drift
restrict default kod limited nomodify nopeer notrap
restrict -6 default kod limited nomodify nopeer notrap
restrict source kod limited nomodify notrap
interface ignore all
interface ignore wildcard
interface listen ix0.100
interface listen igc0
interface listen ix0.10
interface listen ix0
interface listen igc2

[johnpoz]

@conover that is odd for sure, clearly from you sniff looks like you got an answer..

I can't seem to duplicate it though, when on peer it never works, but set server and bam start seeing reach count up.

For grins! could you remove the pool and just point to your server..

[conover]

@johnpoz said in 23.09 Local NTP server "unrechable" (worked with 23.05.1):

@conover that is odd for sure, clearly from you sniff looks like you got an answer..

yes, absolutely, also ntpdate works and synchronizes the clock with the server

For grins! could you remove the pool and just point to your server..

same result

Is there an easy way to go back to 23.05? I would try if it's still working with that. If yes, it must be something with the 23.09 release....

[stephenw10]

If you installed ZFS there will be a BE snap from before the upgrade you can roll back to.

[johnpoz]

@conover said in 23.09 Local NTP server "unrechable" (worked with 23.05.1):

it must be something with the 23.09 release....

While I don't deny you have something wrong - I don't see how its something wrong with 23.09 in general. I point to a local ntp server, I am running 23.09 and not having any issues.

edit:
What is odd, is your status shows type s, pretty sure that means peer (symmetric).. while u would be normal if just pointing to a server - see mine from above shows u.. If I set mine to peer, it doesn't work and it changes to s, and if I look in the xml (do a backup download of your whole config) then open in your fav text editor and look for ntpd

As soon as change mine back to server, it starts working right away. See how the reach started counting and my type is u not s

edit:
https://docs.netgate.com/pfsense/en/latest/monitoring/status/ntp.html

[conover]

@johnpoz Thanks for your ongoing support!

Not saying it's something wrong generally in 23.09 but at least something specific :-) Either in combination with my NTP server or something went wrong during the upgrade.

Went back to 23.05.01 and everything is OK again

After being back to 23.09 same as before...

The flag "u" and "s" appears randomly or changes after some time, currently it is set to "u":