NTP Issue
-
@ARAMP1
Well, it's taken me quite a few hours to get to this post.
I went to check my time precision this morning and noticed that the graphs were all way out.
Checking when it started, it was right after I upgraded to 2.7.2.
Looking at my local Stratum 1 NTP server, it's pending/unreachable and the firewall has fallen back to time.apple.comWith the basics checked, then ACL's adjusted and even firewall rules put in, the ntpupdate -4 command showed that everything is working as it should. (And likely was before I did anything other than upgrade...)
pfsense just refuses to accept any NTP traffic from the LAN side of the house.
Right when I was starting to think it couldn't be much else than a system issue, I found this thread.
As well as confirming that some of us take time seriously, it was nice to know that we've got the same issue.Looks like a roll back is in order, or just wait for the patch
-
Looks like a roll back is in order, or just wait for the patch
Even though there are issues with this (and also the ARP table showing permanent entries), I've decided to keep it at 2.7.2. I still show a stratum 1 using the nist.gov clocks. The only issue would be if the internet goes down. If it becomes an issue with individual machines, I'll direct them to my local NTP server then.
-
just to pile on one more, I have the same issue on 23.09.1 on two 6100s with a local TM2000 on the same lan. I tried all of the things mentioned in these threads with no changes. It looks like it broke on 23.09 for me as well (after reviewing the RRD after I noticed the issue).
-
@JonathanLee said in NTP Issue:
They have a Redmine open for NTP authentication issues I think the patch that Marcos and Matthew merged might also fix this.
I just played around a bit with this and couldn't get it to work.
-
@JonathanLee said in NTP Issue:
They have a Redmine open for NTP authentication issues I think the patch that Marcos and Matthew merged might also fix this.
I just played around a bit with this and couldn't get it to work.
it is not fixed, also not able to get it to work in 24.03
-
Seems the newer versions of software function by way of a different pair of spectacles. I have a number of issues versus older reliable versions. I can tell you I personally like the old software better, it functions and feels far more resilient. I wonder why, I really do. I shouldn’t feel less trust with the new software. Again I hate to admit it’s starting to feel like that. Leading to, I shouldn’t feel afraid to upgrade. I use to get excited about even upgrade, now I question what is going to be broken if I upgrade. I am fearful of updates. Weird right? Is it big tech influence? Who knows, maybe I am just paranoid and don’t want to risk it now that I have it working way I want. Sorry it isn’t working
-
@JonathanLee said in NTP Issue:
Seems the newer versions of software function by way of a different pair of spectacles. I have a number of issues versus older reliable versions. I can tell you I personally like the old software better, it functions and feels far more resilient. I wonder why, I really do. I shouldn’t feel less trust with the new software. Again I hate to admit it’s starting to feel like that. Leading to, I shouldn’t feel afraid to upgrade. I use to get excited about even upgrade, now I question what is going to be broken if I upgrade. I am fearful of updates. Weird right? Is it big tech influence? Who knows, maybe I am just paranoid and don’t want to risk it now that I have it working way I want. Sorry it isn’t working
Fully agreed!
That's the reason why I did not upgraded all my boxes yet.
Firewall software with a bad gut feeling is a "no go" (for me).
So I am started to thinking about changing the system with the next (hardware) cycle.... -
-
So I am started to thinking about changing the system with the next (hardware) cycle....
What you are thinking to go with?
OPNSense, OpenWRT, RouterOS, VyOS, ClearOS, Untangle,......To be honest, not sure yet. Just started thinking about it....
-
To be honest, not sure yet. Just started thinking about it....
I ordered some parts for setting up a tiny LAN server to free pfSense from
some tasks or services. Its nearly 200ish € but its worth in my eyes.RaspBerry PI 4 B (8GB)
GPS hat (u.blox M8)
Micro SD card 64 GB
mSATA board x857
active magnetic GSP antenna (5 meter cable)
USB C - PSULater it should run the following services;
- PI hole or CrowdSec
- syslog-ng server
- NTP (Internet, RTC, GPS)
- RadiusServer (FreeRadius)
- LDAP server (OpenLDAP)
Eventually later a DCF77 kit over USB but this is actually only brainstorming.
So for the entire network it will be then a Stratum1 NTP server with RTC,
GPS and over the internet with 4 pools and 4 servers. -
I just played around a bit with this and couldn't get it to work.
And what did you play with? Can we see your config? Can we see your ntp log? Can we see a simple sniff, you say it works via just using ntp out on the internet. Can we see sniff of that working?
I have been running a pi as my ntp server via a gps hat and yeah it would use pps as the source. For years, multiple versions of pfsense back to pre + days, And currently have a 2.7.2 pfsense as well as 23.09.1 and not having any issues syncing with my pi ntp server.
Are you trying to use auth? To your ntp - you say none of your other devices have any issues syncing with your ntp server? Can we see this output?
example
$ ntpq ntpq> pe remote refid st t when poll reach delay offset jitter ============================================================================== *ntp.home.arpa .PPS. 1 u 57 128 377 1.542 +0.376 0.193 ntpq>My windows box sync with my pi, here is pfsense
[23.09.1-RELEASE][admin@sg4860.home.arpa]/root: ntpq ntpq> pe remote refid st t when poll reach delay offset jitter ============================================================================== *ntp.home.arpa .PPS. 1 u 137 512 377 0.449 -0.741 0.189 ntpq>What are you using as your ntp server, ntpsec, chrony, just the plain jane ntp from ntp.org?
If there was some inherent bug/issue with ntp sync then you would think there would be many posts about it.. Since there does not seem to be - this points to something unique in your setup that is problematic, etc..
here is my settings, where 192.168.3.32 is my pi ntp box.. Which is running ntpsec
Here is my pi output
pi@pi-ntp:~ $ ntpq ntpq> pe remote refid st t when poll reach delay offset jitter ======================================================================================================= *SHM(1) .PPS. 0 l 2 16 377 0.0000 -0.6487 0.0136 -SHM(0) .GPS. 0 l 42 64 377 0.0000 -150.638 1.3091 +ntp-0.gw.illinois.edu 200.7.224.32 2 u 6 64 377 12.1672 2.2484 1.0747 +ntp.your.org .GPS. 1 u 41 64 377 10.5027 2.4140 0.7471 -tock.usshc.com .WAAS. 1 u 37 64 377 16.0153 1.7570 0.6719 -ntp1.netwrx1.com 24.196.111.36 2 u 33 64 377 37.7684 -2.0801 1.9852 ntpq> version ntpsec-1.2.1+85-g1a7bb2e3a ntpq>edit: my 2.7.2 box was off, just booted it - and can see right away that it can talk to my ntp server.. as reach is counting up
edit2: just booted my 24.03 and pointed it to my ntp server and bam its talking to it as you can see reach is counting up as well
here is a pcap of pfsense 3.253 talking to the ntp server 3.32
An intelligent man is sometimes forced to be drunk to spend time with his fools
If you get confused: Listen to the Music Play
Please don't Chat/PM me for help, unless mod related
SG-4860 24.11 | Lab VMs 2.8, 24.11 -
If there was some inherent bug/issue with ntp sync then you would think there would be many posts about it.. Since there does not seem to be - this points to something unique in your setup that is problematic, etc..
well, there are 3 or 4 post addressing that issue and a few people hopped on each thread.
The problem is, that NTP works but not with all NTP servers. The latter one is the issue. So it is not an issue with NTP itself.
We went thru all config options and other options (e.g. delete config and start over again etc) in the named threads.The main point is, NTP worked with the mentioned specifc server type (which are also working with all other clients) in 23.05 / 23.05.01 and stopped working in 23.09 (NTP works again going back to 23.05). And still not works with 24.03.
-
well, there are 3 or 4 post addressing that issue and a few people hopped on each thread.
Lets see out of how many 1000's and 10s of 1000's of pfsense deployments? I am on here way too much, and I am not seeing these posts..
There may well be a bug/issue - but without some actual specifics of the users setup to find something common, users chiming in with me too.. Are of zero help.. For all we know they have some simple misconfiguration.. Like wrong IP address of their ntp server for gosh sake..
the only way to track down what your problem is with info.. Not I play with some stuff and couldn't get it to work ;)
-
@johnpoz guess, all infos can be found here: https://forum.netgate.com/topic/184170/23-09-local-ntp-server-unrechable-worked-with-23-05-1
another thread (besides this one) is here (in German) https://forum.netgate.com/post/1137308
-
I just played around a bit with this and couldn't get it to work.
And what did you play with? Can we see your config? Can we see your ntp log? Can we see a simple sniff, you say it works via just using ntp out on the internet. Can we see sniff of that working?
I have been running a pi as my ntp server via a gps hat and yeah it would use pps as the source. For years, multiple versions of pfsense back to pre + days, And currently have a 2.7.2 pfsense as well as 23.09.1 and not having any issues syncing with my pi ntp server.
Are you trying to use auth? To your ntp - you say none of your other devices have any issues syncing with your ntp server? Can we see this output?
example
$ ntpq ntpq> pe remote refid st t when poll reach delay offset jitter ============================================================================== *ntp.home.arpa .PPS. 1 u 57 128 377 1.542 +0.376 0.193 ntpq>My windows box sync with my pi, here is pfsense
[23.09.1-RELEASE][admin@sg4860.home.arpa]/root: ntpq ntpq> pe remote refid st t when poll reach delay offset jitter ============================================================================== *ntp.home.arpa .PPS. 1 u 137 512 377 0.449 -0.741 0.189 ntpq>What are you using as your ntp server, ntpsec, chrony, just the plain jane ntp from ntp.org?
If there was some inherent bug/issue with ntp sync then you would think there would be many posts about it.. Since there does not seem to be - this points to something unique in your setup that is problematic, etc..
here is my settings, where 192.168.3.32 is my pi ntp box.. Which is running ntpsec
Here is my pi output
pi@pi-ntp:~ $ ntpq ntpq> pe remote refid st t when poll reach delay offset jitter ======================================================================================================= *SHM(1) .PPS. 0 l 2 16 377 0.0000 -0.6487 0.0136 -SHM(0) .GPS. 0 l 42 64 377 0.0000 -150.638 1.3091 +ntp-0.gw.illinois.edu 200.7.224.32 2 u 6 64 377 12.1672 2.2484 1.0747 +ntp.your.org .GPS. 1 u 41 64 377 10.5027 2.4140 0.7471 -tock.usshc.com .WAAS. 1 u 37 64 377 16.0153 1.7570 0.6719 -ntp1.netwrx1.com 24.196.111.36 2 u 33 64 377 37.7684 -2.0801 1.9852 ntpq> version ntpsec-1.2.1+85-g1a7bb2e3a ntpq>edit: my 2.7.2 box was off, just booted it - and can see right away that it can talk to my ntp server.. as reach is counting up
edit2: just booted my 24.03 and pointed it to my ntp server and bam its talking to it as you can see reach is counting up as well
here is a pcap of pfsense 3.253 talking to the ntp server 3.32
Oh, I'd bet it's 100% user error, just want to figure out where.
It appears I have it set up like yours.
I'm using one of those chinese ebay boxes. I've had it running fine in pfSense for about 3 years now.
Here are the logs related to the server at 192.168.10.120
Jun 2 12:45:34 ntpd 8952 192.168.10.120 local addr 192.168.10.1 -> <null>
Jun 2 12:06:10 ntpd 15912 192.168.10.120 local addr 192.168.10.1 -> <null>
Jun 2 12:04:11 ntpd 15912 192.168.10.120 8011 81 mobilize assoc 17918
Jun 2 12:45:34 ntpd 75278 192.168.10.120 8011 81 mobilize assoc 23975
Jun 2 12:06:21 ntpd 8952 192.168.10.120 8011 81 mobilize assoc 59753
Jun 2 12:06:10 ntpd 15912 192.168.10.120 8012 82 demobilize assoc 17918
Jun 2 12:45:34 ntpd 8952 192.168.10.120 8012 82 demobilize assoc 59753 -
@ARAMP1 one thing I see as different is why are you pointing to multiple pools that are really the same? You run your own ntp server, while sure it should point to some others vs just its gps time.. pfsense really has litte need other than just pointing to your one ntp server that you are running.
Why can it not talk to any of those other ntp servers either.. All of those other ones should be available via wan, that it is seems it talking to the 128.138.141.172 box.
You cut off your config - are you trying to do auth?
If you want to add more later - sure but for troubleshooting I would just point to your 1 local server.. And then sniff do you see an answer?
Can we see your other devices that are syncing off this local server? For example I already showed my windows box syncing - here is one of my switches using it
sg300-28#sho sntp status Clock is synchronized, stratum 1, reference is 192.168.3.32, unicast Unicast servers: Server : 192.168.3.32 Source : Static Stratum : 1 Status : up Last Response : 17:03:25.0 web Jun 2 2024 Offset : 347.3516068 mSec Delay : -1000 mSechere is my nas, and printer all pointing to my local ntp - showing working
For troubleshooting - I would set the only interface on to be the one interface where your ntp server is. And I would sniff the traffic to validate you see it sending to it, and getting a response on its own accord, not with the ntpdate command, etc.
Here is my unifi controller setting, and showing that one of the AP is sync'd etc
edit: oh my nas is using old name, which still resolves - but should change that to home.arpa vs local.lan
What are you running for your ntp server? ntp, ntpsec, chrony?
edit2: that you are only able to talk to 1 of the many many ntp servers your pointing to is very odd to me.. If there was something wrong with pfsense, why is talking to anything. If there was some disconnect between pfsense and your ntp server, why is it not talking to all the other servers your pointing to?
Clearly something is amiss - figuring out what it is the puzzle needed to be solved.
-
@ARAMP1 one thing I see as different is why are you pointing to multiple pools that are really the same? You run your own ntp server, while sure it should point to some others vs just its gps time.. pfsense really has litte need other than just pointing to your one ntp server that you are running.
Why can it not talk to any of those other ntp servers either.. All of those other ones should be available via wan, that it is seems it talking to the 128.138.141.172 box.
You cut off your config - are you trying to do auth?
If you want to add more later - sure but for troubleshooting I would just point to your 1 local server.. And then sniff do you see an answer?
Can we see your other devices that are syncing off this local server? For example I already showed my windows box syncing - here is one of my switches using it
sg300-28#sho sntp status Clock is synchronized, stratum 1, reference is 192.168.3.32, unicast Unicast servers: Server : 192.168.3.32 Source : Static Stratum : 1 Status : up Last Response : 17:03:25.0 web Jun 2 2024 Offset : 347.3516068 mSec Delay : -1000 mSechere is my nas, and printer all pointing to my local ntp - showing working
For troubleshooting - I would set the only interface on to be the one interface where your ntp server is. And I would sniff the traffic to validate you see it sending to it, and getting a response on its own accord, not with the ntpdate command, etc.
Here is my unifi controller setting, and showing that one of the AP is sync'd etc
edit: oh my nas is using old name, which still resolves - but should change that to home.arpa vs local.lan
What are you running for your ntp server? ntp, ntpsec, chrony?
edit2: that you are only able to talk to 1 of the many many ntp servers your pointing to is very odd to me.. If there was something wrong with pfsense, why is talking to anything. If there was some disconnect between pfsense and your ntp server, why is it not talking to all the other servers your pointing to?
Clearly something is amiss - figuring out what it is the puzzle needed to be solved.
Appreciate the reply.
I'm not doing NTPv3 authentication.
I actually originally had it only pointing to my local server, but when it started acting up, I started adding. I assumed that since I checked "prefer" it would default to that one if it started working.
Here's what I get when I delete all the others.
I can point anything on the network to my NTP and it seems to work. Here's the windows 10 machine I'm on now.
Here's a TrueNAS server that I just pointed it to.
Really, I only need it for my Blue Iris machine because I want all of my security cameras synced. Other than that, it's just nice to have it working.
Forgive me, but I'm not sure how do do the ole sniff.
-
@ARAMP1 the sniff would be just packet capture under the diagnostic menu.. You can have it only capture the ntp from your IP of your ntp server and port 123.
Select the interface used to talk to your ntp server, assuming that is your lan interface.
You can then download that capture and we can look at it with say wireshark.
The info from your nas is good.. what IP is your nas on, I assume its on this same 192.168.10 network.
edit: don't camera's sync their time via ipc?
I don't have ntp setup on my cameras - they sync with the nvr via ipc.. But blue iris could be different for sure. The nvr syncs directly with my ntp server.. Why wouldn't you sync your blue iris and your cameras directly with your ntp server? vs pfsense?
That being said - pfsense for sure should be able to sync with your ntp server.. But if you can just sync your camera stuff with your ntp server, then the pfsense time sync isn't as critical, etc. And can figure out what is going on it with it without having to worry about your camera time being off, etc.
-
@ARAMP1 the sniff would be just packet capture under the diagnostic menu.. You can have it only capture the ntp from your IP of your ntp server and port 123.
Select the interface used to talk to your ntp server, assuming that is your lan interface.
You can then download that capture and we can look at it with say wireshark.
The info from your nas is good.. what IP is your nas on, I assume its on this same 192.168.10 network.
My windows machine is on the 10.10.50 network and that truenas machine is on the 192.168.60 network but I have rules on each of those VLANs to access 192.168.10.120 for NTP (the 192.168.10 network is my management network with all my switches, access points, etc).
For cameras, here's the rules I have in place.
-
@ARAMP1 well that is interesting.. So the devices on different networks sync fine - but the pfsense on the same network is not able to reach it??
Your not doing anything with ethernet filtering, nor any floating rules for outbound filtering?
As a test - your truenas can act as ntp server I would assume.. Can you point pfsense to it?
Example - just pointed pfsense to my nas.
notice it shows the 9.10 address of my nas, and it shows its ref as my pi ntp server the 3.32 IP
An intelligent man is sometimes forced to be drunk to spend time with his fools
If you get confused: Listen to the Music Play
Please don't Chat/PM me for help, unless mod related
SG-4860 24.11 | Lab VMs 2.8, 24.11
