Unable to synch with internal network NTP server

dennypage

@RonRN18 questions that come to mind:

is the time server using a fixed address?
in the pfSense config, are you referring to the time server by ip address or hostname?
does reachability for the time server initially go completely to 377 before descending to 0? Or are there gaps?
do you have other time servers configured in pfSense?
have you watched the interaction with tcpdump?

johnpoz

@RonRN18 up to a minutes - you mean you get a reach of 1, it will take a few answers for reach to get to 377. I don't really see it counting up to 377 in a minute.

RonRN18

@dennypage

The time server is using a fixed IP address; 172.18.0.100
list itemI am referring to the time server's IP in the pfSense config
The reachability slowly climbs up to about 377 and then immediately drops to zero
I have another internal time server using a Pi (172.18.0.10), but after changing its GPS receiver, I haven't gotten it back up and functioning as a Stratum 1, so it is connecting to another public NTP and showing up as a Stratum 3. It is also using Chrony as its time server. This one shows up with a reachability of 377 but BOTH of them show "Unreach/Pending" under status.
As of yet, I have not "watched the interaction with tcpdump", not exactly sure how or what specifically I'd be looking for.

RonRN18

@johnpoz

I know it isn't a riveting video, but I created a relatively short video of the NTP status page. I hit the restart of the pfSense NTP server about 4 seconds into the video and the overall is about 1 minute and 17 seconds before 172.18.0.100 drops to a reachability of zero; I guess it goes to zero after 200, not 377.

Google Drive Video of NTP status

dennypage

@RonRN18 Reach is a bit mask. The sequence 1,2,4,10,20,40,100,200,0 indicates you are only getting one exchange.

There is a loss of connection between pfSense and your time server. I would check with tcpdump on both ends.

johnpoz

@RonRN18 as @dennypage pointed out.. That reach value is a bit shift register.. Normally if you were seeing responses for each query the count up would be

0, 1, 3, 7, 17, 37, 77, 177, 377

Your values show 1st answer, but then the register shifting with subsequent no answers to your next query.

So you would have

00000001 = 1
00000010 = 2
00000100 = 4
00001000 = 10
00010000 = 20
00100000 = 40
01000000 = 100
10000000 = 200

Where you last received answer moves through the register

where if you were getting responses your count would go

00000001 = 1
00000011 = 3
00000111 = 7
00001111 = 17
00011111 = 37
00111111 = 77
01111111 = 177
11111111 = 377

You can use the calc in windows to help, change it to programmer, set your input to octal, then look at the binary value.

dennypage

Thanks for the added explanation @johnpoz. It’s hard to write detailed posts from a phone.

Btw @RonRN18 , I would also check system and filter logs on pfSense for completeness.

johnpoz

@dennypage no problem - I thought showing them the sequence in both directions would be helpful in the lightbulb going off.

Should of prob shown a case where you miss one and and its in the middle of the register.

Its actually pretty slick once it clicks on how the numbers work out.

edit: say for example you were just humming along and you missed one.. and then the next 3 were good, your reach would show 367, and if you put that in binary format you can see exactly which one you missed out of the last 8

RonRN18

@dennypage

I went through the tcpdump from both the time server and pfSense and ran them through ChatGPT to see if they could identify any issues (like looking for a needle in a haystack for a human). It responded that all looked fine. Below is a screenshot of my Status -> System Logs -> NTP:

alt text

I see a bunch of reachable, followed by unreachable. I'm not sure why this is.

dennypage

@RonRN18 I meant check the System and Firewall (filter) logs.

FWIW, I would not expect ChatGPT to be useful in analyzing packet traces. Needs a human. Use “tcpdump port ntp” and see if the packets appear on both sides. Post both if you’re unsure.

RonRN18

@RonRN18

I went back into the Services -> NTP section and I see I had added 172.18.0.100 as a "peer". I just changed it to a "server". I thought this worked at first but it soon failed. I also had 172.18.0.10 listed as a "peer" and changed it to "server". This seems to have fixed the issue.

alt text

One Stratum 1 server fixed... now on to the other. 172.18.0.10 was my original but I borked it somehow. It is now just functioning as a Stratum 3 until I fix it.

johnpoz

@RonRN18 you prob not going to want your ntp servers as a peer to your pfsense. The server you point to in pfsense should be set as server. Yes it will show as active peer in the status.

Pointing to 2 seems counter productive.. It should be just 1 or at least 3.

Also - clear your freaking notifications.. 769 of them? Come on!

Also a poll of 8 seconds seems a bit excessive if you ask me ;)

If your 0.10 box is syncing to some public ntp server or pool.. It will never be stratum 1.. Best it could ever be is 2, and that is if you were syncing to a 1..

dennypage

@johnpoz said in Unable to synch with internal network NTP server:

Also a poll of 8 seconds seems a bit excessive if you ask me ;)

Not for a LAN based server. I have done a lot of hw ntp testing in the past. I use -4 (16 per second) for my main servers. Unfortunately, ntpd cannot operate below 3 (8 seconds per).