NTP Issue

JonathanLee

They have a Redmine open for NTP authentication issues I think the patch that Marcos and Matthew merged might also fix this. The GitHub was merged a couple days ago but has not been looked at but it looked to incorporate many fixes not just authentication issues.

https://redmine.pfsense.org/issues/8794

https://github.com/pfsense/pfsense/commit/847e417b5612f28bc1e84ca028a980df9c5c57a7

admacdo

@ARAMP1
Well, it's taken me quite a few hours to get to this post.
I went to check my time precision this morning and noticed that the graphs were all way out.
Checking when it started, it was right after I upgraded to 2.7.2.
Looking at my local Stratum 1 NTP server, it's pending/unreachable and the firewall has fallen back to time.apple.com

With the basics checked, then ACL's adjusted and even firewall rules put in, the ntpupdate -4 command showed that everything is working as it should. (And likely was before I did anything other than upgrade...)
pfsense just refuses to accept any NTP traffic from the LAN side of the house.
Right when I was starting to think it couldn't be much else than a system issue, I found this thread.
As well as confirming that some of us take time seriously, it was nice to know that we've got the same issue.

Looks like a roll back is in order, or just wait for the patch

ARAMP1

@admacdo said in NTP Issue:

Looks like a roll back is in order, or just wait for the patch

Even though there are issues with this (and also the ARP table showing permanent entries), I've decided to keep it at 2.7.2. I still show a stratum 1 using the nist.gov clocks. The only issue would be if the internet goes down. If it becomes an issue with individual machines, I'll direct them to my local NTP server then.

EricMc1

just to pile on one more, I have the same issue on 23.09.1 on two 6100s with a local TM2000 on the same lan. I tried all of the things mentioned in these threads with no changes. It looks like it broke on 23.09 for me as well (after reviewing the RRD after I noticed the issue).

ARAMP1

@JonathanLee said in NTP Issue:

They have a Redmine open for NTP authentication issues I think the patch that Marcos and Matthew merged might also fix this.

I just played around a bit with this and couldn't get it to work.

conover

@ARAMP1 said in NTP Issue:

@JonathanLee said in NTP Issue:

They have a Redmine open for NTP authentication issues I think the patch that Marcos and Matthew merged might also fix this.

I just played around a bit with this and couldn't get it to work.

it is not fixed, also not able to get it to work in 24.03

JonathanLee

Seems the newer versions of software function by way of a different pair of spectacles. I have a number of issues versus older reliable versions. I can tell you I personally like the old software better, it functions and feels far more resilient. I wonder why, I really do. I shouldn’t feel less trust with the new software. Again I hate to admit it’s starting to feel like that. Leading to, I shouldn’t feel afraid to upgrade. I use to get excited about even upgrade, now I question what is going to be broken if I upgrade. I am fearful of updates. Weird right? Is it big tech influence? Who knows, maybe I am just paranoid and don’t want to risk it now that I have it working way I want. Sorry it isn’t working

conover

@JonathanLee said in NTP Issue:

Seems the newer versions of software function by way of a different pair of spectacles. I have a number of issues versus older reliable versions. I can tell you I personally like the old software better, it functions and feels far more resilient. I wonder why, I really do. I shouldn’t feel less trust with the new software. Again I hate to admit it’s starting to feel like that. Leading to, I shouldn’t feel afraid to upgrade. I use to get excited about even upgrade, now I question what is going to be broken if I upgrade. I am fearful of updates. Weird right? Is it big tech influence? Who knows, maybe I am just paranoid and don’t want to risk it now that I have it working way I want. Sorry it isn’t working

Fully agreed!
That's the reason why I did not upgraded all my boxes yet.
Firewall software with a bad gut feeling is a "no go" (for me).
So I am started to thinking about changing the system with the next (hardware) cycle....

Dobby_

@conover said in NTP Issue:

So I am started to thinking about changing the system with the next (hardware) cycle....

What you are thinking to go with?
OPNSense, OpenWRT, RouterOS, VyOS, ClearOS, Untangle,......

conover

@Dobby_ said in NTP Issue:

@conover said in NTP Issue:

So I am started to thinking about changing the system with the next (hardware) cycle....

What you are thinking to go with?
OPNSense, OpenWRT, RouterOS, VyOS, ClearOS, Untangle,......

To be honest, not sure yet. Just started thinking about it....

Dobby_

@conover said in NTP Issue:

To be honest, not sure yet. Just started thinking about it....

I ordered some parts for setting up a tiny LAN server to free pfSense from
some tasks or services. Its nearly 200ish € but its worth in my eyes.

RaspBerry PI 4 B (8GB)
GPS hat (u.blox M8)
Micro SD card 64 GB
mSATA board x857
active magnetic GSP antenna (5 meter cable)
USB C - PSU

Later it should run the following services;

PI hole or CrowdSec
syslog-ng server
NTP (Internet, RTC, GPS)
RadiusServer (FreeRadius)
LDAP server (OpenLDAP)

Eventually later a DCF77 kit over USB but this is actually only brainstorming.
So for the entire network it will be then a Stratum1 NTP server with RTC,
GPS and over the internet with 4 pools and 4 servers.

johnpoz

@ARAMP1 said in NTP Issue:

I just played around a bit with this and couldn't get it to work.

And what did you play with? Can we see your config? Can we see your ntp log? Can we see a simple sniff, you say it works via just using ntp out on the internet. Can we see sniff of that working?

I have been running a pi as my ntp server via a gps hat and yeah it would use pps as the source. For years, multiple versions of pfsense back to pre + days, And currently have a 2.7.2 pfsense as well as 23.09.1 and not having any issues syncing with my pi ntp server.

Are you trying to use auth? To your ntp - you say none of your other devices have any issues syncing with your ntp server? Can we see this output?

example

$ ntpq
ntpq> pe
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*ntp.home.arpa   .PPS.            1 u   57  128  377    1.542   +0.376   0.193
ntpq>

My windows box sync with my pi, here is pfsense

[23.09.1-RELEASE][admin@sg4860.home.arpa]/root: ntpq
ntpq> pe
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*ntp.home.arpa   .PPS.            1 u  137  512  377    0.449   -0.741   0.189
ntpq>

What are you using as your ntp server, ntpsec, chrony, just the plain jane ntp from ntp.org?

If there was some inherent bug/issue with ntp sync then you would think there would be many posts about it.. Since there does not seem to be - this points to something unique in your setup that is problematic, etc..

here is my settings, where 192.168.3.32 is my pi ntp box.. Which is running ntpsec

Here is my pi output

pi@pi-ntp:~ $ ntpq
ntpq> pe
     remote                                   refid      st t when poll reach   delay   offset   jitter
=======================================================================================================
*SHM(1)                                  .PPS.            0 l    2   16  377   0.0000  -0.6487   0.0136
-SHM(0)                                  .GPS.            0 l   42   64  377   0.0000 -150.638   1.3091
+ntp-0.gw.illinois.edu                   200.7.224.32     2 u    6   64  377  12.1672   2.2484   1.0747
+ntp.your.org                            .GPS.            1 u   41   64  377  10.5027   2.4140   0.7471
-tock.usshc.com                          .WAAS.           1 u   37   64  377  16.0153   1.7570   0.6719
-ntp1.netwrx1.com                        24.196.111.36    2 u   33   64  377  37.7684  -2.0801   1.9852
ntpq> version
ntpsec-1.2.1+85-g1a7bb2e3a
ntpq>

edit: my 2.7.2 box was off, just booted it - and can see right away that it can talk to my ntp server.. as reach is counting up

edit2: just booted my 24.03 and pointed it to my ntp server and bam its talking to it as you can see reach is counting up as well

here is a pcap of pfsense 3.253 talking to the ntp server 3.32

conover

@johnpoz said in NTP Issue:

If there was some inherent bug/issue with ntp sync then you would think there would be many posts about it.. Since there does not seem to be - this points to something unique in your setup that is problematic, etc..

well, there are 3 or 4 post addressing that issue and a few people hopped on each thread.

The problem is, that NTP works but not with all NTP servers. The latter one is the issue. So it is not an issue with NTP itself.
We went thru all config options and other options (e.g. delete config and start over again etc) in the named threads.

The main point is, NTP worked with the mentioned specifc server type (which are also working with all other clients) in 23.05 / 23.05.01 and stopped working in 23.09 (NTP works again going back to 23.05). And still not works with 24.03.

johnpoz

@conover said in NTP Issue:

well, there are 3 or 4 post addressing that issue and a few people hopped on each thread.

Lets see out of how many 1000's and 10s of 1000's of pfsense deployments? I am on here way too much, and I am not seeing these posts..

There may well be a bug/issue - but without some actual specifics of the users setup to find something common, users chiming in with me too.. Are of zero help.. For all we know they have some simple misconfiguration.. Like wrong IP address of their ntp server for gosh sake..

the only way to track down what your problem is with info.. Not I play with some stuff and couldn't get it to work ;)

conover

@johnpoz guess, all infos can be found here: https://forum.netgate.com/topic/184170/23-09-local-ntp-server-unrechable-worked-with-23-05-1

another thread (besides this one) is here (in German) https://forum.netgate.com/post/1137308

ARAMP1

@johnpoz said in NTP Issue:

@ARAMP1 said in NTP Issue:

I just played around a bit with this and couldn't get it to work.

And what did you play with? Can we see your config? Can we see your ntp log? Can we see a simple sniff, you say it works via just using ntp out on the internet. Can we see sniff of that working?

I have been running a pi as my ntp server via a gps hat and yeah it would use pps as the source. For years, multiple versions of pfsense back to pre + days, And currently have a 2.7.2 pfsense as well as 23.09.1 and not having any issues syncing with my pi ntp server.

Are you trying to use auth? To your ntp - you say none of your other devices have any issues syncing with your ntp server? Can we see this output?

example
$ ntpq
ntpq> pe
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*ntp.home.arpa   .PPS.            1 u   57  128  377    1.542   +0.376   0.193
ntpq>
My windows box sync with my pi, here is pfsense
[23.09.1-RELEASE][admin@sg4860.home.arpa]/root: ntpq
ntpq> pe
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*ntp.home.arpa   .PPS.            1 u  137  512  377    0.449   -0.741   0.189
ntpq> 
What are you using as your ntp server, ntpsec, chrony, just the plain jane ntp from ntp.org?

If there was some inherent bug/issue with ntp sync then you would think there would be many posts about it.. Since there does not seem to be - this points to something unique in your setup that is problematic, etc..

here is my settings, where 192.168.3.32 is my pi ntp box.. Which is running ntpsec

Here is my pi output
pi@pi-ntp:~ $ ntpq
ntpq> pe
     remote                                   refid      st t when poll reach   delay   offset   jitter
=======================================================================================================
*SHM(1)                                  .PPS.            0 l    2   16  377   0.0000  -0.6487   0.0136
-SHM(0)                                  .GPS.            0 l   42   64  377   0.0000 -150.638   1.3091
+ntp-0.gw.illinois.edu                   200.7.224.32     2 u    6   64  377  12.1672   2.2484   1.0747
+ntp.your.org                            .GPS.            1 u   41   64  377  10.5027   2.4140   0.7471
-tock.usshc.com                          .WAAS.           1 u   37   64  377  16.0153   1.7570   0.6719
-ntp1.netwrx1.com                        24.196.111.36    2 u   33   64  377  37.7684  -2.0801   1.9852
ntpq> version
ntpsec-1.2.1+85-g1a7bb2e3a
ntpq>
edit: my 2.7.2 box was off, just booted it - and can see right away that it can talk to my ntp server.. as reach is counting up

edit2: just booted my 24.03 and pointed it to my ntp server and bam its talking to it as you can see reach is counting up as well

here is a pcap of pfsense 3.253 talking to the ntp server 3.32

Oh, I'd bet it's 100% user error, just want to figure out where.

It appears I have it set up like yours.

NTP 3.jpg NTP 2.jpg NTP 1.jpg

I'm using one of those chinese ebay boxes. I've had it running fine in pfSense for about 3 years now.

Here are the logs related to the server at 192.168.10.120

Jun 2 12:45:34 ntpd 8952 192.168.10.120 local addr 192.168.10.1 -> <null>
Jun 2 12:06:10 ntpd 15912 192.168.10.120 local addr 192.168.10.1 -> <null>
Jun 2 12:04:11 ntpd 15912 192.168.10.120 8011 81 mobilize assoc 17918
Jun 2 12:45:34 ntpd 75278 192.168.10.120 8011 81 mobilize assoc 23975
Jun 2 12:06:21 ntpd 8952 192.168.10.120 8011 81 mobilize assoc 59753
Jun 2 12:06:10 ntpd 15912 192.168.10.120 8012 82 demobilize assoc 17918
Jun 2 12:45:34 ntpd 8952 192.168.10.120 8012 82 demobilize assoc 59753

johnpoz

@ARAMP1 one thing I see as different is why are you pointing to multiple pools that are really the same? You run your own ntp server, while sure it should point to some others vs just its gps time.. pfsense really has litte need other than just pointing to your one ntp server that you are running.

Why can it not talk to any of those other ntp servers either.. All of those other ones should be available via wan, that it is seems it talking to the 128.138.141.172 box.

You cut off your config - are you trying to do auth?

If you want to add more later - sure but for troubleshooting I would just point to your 1 local server.. And then sniff do you see an answer?

Can we see your other devices that are syncing off this local server? For example I already showed my windows box syncing - here is one of my switches using it

sg300-28#sho sntp status

Clock is synchronized, stratum 1, reference is 192.168.3.32, unicast

Unicast servers:

Server            : 192.168.3.32
  Source          : Static
  Stratum         : 1
  Status          : up
  Last Response   : 17:03:25.0 web Jun 2 2024
  Offset          : 347.3516068 mSec
  Delay           : -1000 mSec

here is my nas, and printer all pointing to my local ntp - showing working

For troubleshooting - I would set the only interface on to be the one interface where your ntp server is. And I would sniff the traffic to validate you see it sending to it, and getting a response on its own accord, not with the ntpdate command, etc.

Here is my unifi controller setting, and showing that one of the AP is sync'd etc

edit: oh my nas is using old name, which still resolves - but should change that to home.arpa vs local.lan

What are you running for your ntp server? ntp, ntpsec, chrony?

edit2: that you are only able to talk to 1 of the many many ntp servers your pointing to is very odd to me.. If there was something wrong with pfsense, why is talking to anything. If there was some disconnect between pfsense and your ntp server, why is it not talking to all the other servers your pointing to?

Clearly something is amiss - figuring out what it is the puzzle needed to be solved.

ARAMP1

@johnpoz said in NTP Issue:

@ARAMP1 one thing I see as different is why are you pointing to multiple pools that are really the same? You run your own ntp server, while sure it should point to some others vs just its gps time.. pfsense really has litte need other than just pointing to your one ntp server that you are running.

Why can it not talk to any of those other ntp servers either.. All of those other ones should be available via wan, that it is seems it talking to the 128.138.141.172 box.

You cut off your config - are you trying to do auth?

If you want to add more later - sure but for troubleshooting I would just point to your 1 local server.. And then sniff do you see an answer?

Can we see your other devices that are syncing off this local server? For example I already showed my windows box syncing - here is one of my switches using it
sg300-28#sho sntp status

Clock is synchronized, stratum 1, reference is 192.168.3.32, unicast

Unicast servers:

Server            : 192.168.3.32
  Source          : Static
  Stratum         : 1
  Status          : up
  Last Response   : 17:03:25.0 web Jun 2 2024
  Offset          : 347.3516068 mSec
  Delay           : -1000 mSec
here is my nas, and printer all pointing to my local ntp - showing working

For troubleshooting - I would set the only interface on to be the one interface where your ntp server is. And I would sniff the traffic to validate you see it sending to it, and getting a response on its own accord, not with the ntpdate command, etc.

Here is my unifi controller setting, and showing that one of the AP is sync'd etc

edit: oh my nas is using old name, which still resolves - but should change that to home.arpa vs local.lan

What are you running for your ntp server? ntp, ntpsec, chrony?

edit2: that you are only able to talk to 1 of the many many ntp servers your pointing to is very odd to me.. If there was something wrong with pfsense, why is talking to anything. If there was some disconnect between pfsense and your ntp server, why is it not talking to all the other servers your pointing to?

Clearly something is amiss - figuring out what it is the puzzle needed to be solved.

Appreciate the reply.

I'm not doing NTPv3 authentication.

I actually originally had it only pointing to my local server, but when it started acting up, I started adding. I assumed that since I checked "prefer" it would default to that one if it started working.

Here's what I get when I delete all the others.

NTP 5.jpg

I can point anything on the network to my NTP and it seems to work. Here's the windows 10 machine I'm on now.

Sync with NTP.jpg

Show NTP.jpg

Here's a TrueNAS server that I just pointed it to.

NTP Truenas.jpg

Really, I only need it for my Blue Iris machine because I want all of my security cameras synced. Other than that, it's just nice to have it working.

Forgive me, but I'm not sure how do do the ole sniff.

johnpoz

@ARAMP1 the sniff would be just packet capture under the diagnostic menu.. You can have it only capture the ntp from your IP of your ntp server and port 123.

Select the interface used to talk to your ntp server, assuming that is your lan interface.

You can then download that capture and we can look at it with say wireshark.

The info from your nas is good.. what IP is your nas on, I assume its on this same 192.168.10 network.

edit: don't camera's sync their time via ipc?

I don't have ntp setup on my cameras - they sync with the nvr via ipc.. But blue iris could be different for sure. The nvr syncs directly with my ntp server.. Why wouldn't you sync your blue iris and your cameras directly with your ntp server? vs pfsense?

That being said - pfsense for sure should be able to sync with your ntp server.. But if you can just sync your camera stuff with your ntp server, then the pfsense time sync isn't as critical, etc. And can figure out what is going on it with it without having to worry about your camera time being off, etc.

ARAMP1

@johnpoz said in NTP Issue:

@ARAMP1 the sniff would be just packet capture under the diagnostic menu.. You can have it only capture the ntp from your IP of your ntp server and port 123.

Select the interface used to talk to your ntp server, assuming that is your lan interface.

You can then download that capture and we can look at it with say wireshark.

The info from your nas is good.. what IP is your nas on, I assume its on this same 192.168.10 network.

My windows machine is on the 10.10.50 network and that truenas machine is on the 192.168.60 network but I have rules on each of those VLANs to access 192.168.10.120 for NTP (the 192.168.10 network is my management network with all my switches, access points, etc).

For cameras, here's the rules I have in place.

Cameras Port 123.jpg