NTP Issue
-
I just played around a bit with this and couldn't get it to work.
And what did you play with? Can we see your config? Can we see your ntp log? Can we see a simple sniff, you say it works via just using ntp out on the internet. Can we see sniff of that working?
I have been running a pi as my ntp server via a gps hat and yeah it would use pps as the source. For years, multiple versions of pfsense back to pre + days, And currently have a 2.7.2 pfsense as well as 23.09.1 and not having any issues syncing with my pi ntp server.
Are you trying to use auth? To your ntp - you say none of your other devices have any issues syncing with your ntp server? Can we see this output?
example
$ ntpq ntpq> pe remote refid st t when poll reach delay offset jitter ============================================================================== *ntp.home.arpa .PPS. 1 u 57 128 377 1.542 +0.376 0.193 ntpq>
My windows box sync with my pi, here is pfsense
[23.09.1-RELEASE][admin@sg4860.home.arpa]/root: ntpq ntpq> pe remote refid st t when poll reach delay offset jitter ============================================================================== *ntp.home.arpa .PPS. 1 u 137 512 377 0.449 -0.741 0.189 ntpq>
What are you using as your ntp server, ntpsec, chrony, just the plain jane ntp from ntp.org?
If there was some inherent bug/issue with ntp sync then you would think there would be many posts about it.. Since there does not seem to be - this points to something unique in your setup that is problematic, etc..
here is my settings, where 192.168.3.32 is my pi ntp box.. Which is running ntpsec
Here is my pi output
pi@pi-ntp:~ $ ntpq ntpq> pe remote refid st t when poll reach delay offset jitter ======================================================================================================= *SHM(1) .PPS. 0 l 2 16 377 0.0000 -0.6487 0.0136 -SHM(0) .GPS. 0 l 42 64 377 0.0000 -150.638 1.3091 +ntp-0.gw.illinois.edu 200.7.224.32 2 u 6 64 377 12.1672 2.2484 1.0747 +ntp.your.org .GPS. 1 u 41 64 377 10.5027 2.4140 0.7471 -tock.usshc.com .WAAS. 1 u 37 64 377 16.0153 1.7570 0.6719 -ntp1.netwrx1.com 24.196.111.36 2 u 33 64 377 37.7684 -2.0801 1.9852 ntpq> version ntpsec-1.2.1+85-g1a7bb2e3a ntpq>
edit: my 2.7.2 box was off, just booted it - and can see right away that it can talk to my ntp server.. as reach is counting up
edit2: just booted my 24.03 and pointed it to my ntp server and bam its talking to it as you can see reach is counting up as well
here is a pcap of pfsense 3.253 talking to the ntp server 3.32
Oh, I'd bet it's 100% user error, just want to figure out where.
It appears I have it set up like yours.
I'm using one of those chinese ebay boxes. I've had it running fine in pfSense for about 3 years now.
Here are the logs related to the server at 192.168.10.120
Jun 2 12:45:34 ntpd 8952 192.168.10.120 local addr 192.168.10.1 -> <null>
Jun 2 12:06:10 ntpd 15912 192.168.10.120 local addr 192.168.10.1 -> <null>
Jun 2 12:04:11 ntpd 15912 192.168.10.120 8011 81 mobilize assoc 17918
Jun 2 12:45:34 ntpd 75278 192.168.10.120 8011 81 mobilize assoc 23975
Jun 2 12:06:21 ntpd 8952 192.168.10.120 8011 81 mobilize assoc 59753
Jun 2 12:06:10 ntpd 15912 192.168.10.120 8012 82 demobilize assoc 17918
Jun 2 12:45:34 ntpd 8952 192.168.10.120 8012 82 demobilize assoc 59753 -
@ARAMP1 one thing I see as different is why are you pointing to multiple pools that are really the same? You run your own ntp server, while sure it should point to some others vs just its gps time.. pfsense really has litte need other than just pointing to your one ntp server that you are running.
Why can it not talk to any of those other ntp servers either.. All of those other ones should be available via wan, that it is seems it talking to the 128.138.141.172 box.
You cut off your config - are you trying to do auth?
If you want to add more later - sure but for troubleshooting I would just point to your 1 local server.. And then sniff do you see an answer?
Can we see your other devices that are syncing off this local server? For example I already showed my windows box syncing - here is one of my switches using it
sg300-28#sho sntp status Clock is synchronized, stratum 1, reference is 192.168.3.32, unicast Unicast servers: Server : 192.168.3.32 Source : Static Stratum : 1 Status : up Last Response : 17:03:25.0 web Jun 2 2024 Offset : 347.3516068 mSec Delay : -1000 mSec
here is my nas, and printer all pointing to my local ntp - showing working
For troubleshooting - I would set the only interface on to be the one interface where your ntp server is. And I would sniff the traffic to validate you see it sending to it, and getting a response on its own accord, not with the ntpdate command, etc.
Here is my unifi controller setting, and showing that one of the AP is sync'd etc
edit: oh my nas is using old name, which still resolves - but should change that to home.arpa vs local.lan
What are you running for your ntp server? ntp, ntpsec, chrony?
edit2: that you are only able to talk to 1 of the many many ntp servers your pointing to is very odd to me.. If there was something wrong with pfsense, why is talking to anything. If there was some disconnect between pfsense and your ntp server, why is it not talking to all the other servers your pointing to?
Clearly something is amiss - figuring out what it is the puzzle needed to be solved.
-
@ARAMP1 one thing I see as different is why are you pointing to multiple pools that are really the same? You run your own ntp server, while sure it should point to some others vs just its gps time.. pfsense really has litte need other than just pointing to your one ntp server that you are running.
Why can it not talk to any of those other ntp servers either.. All of those other ones should be available via wan, that it is seems it talking to the 128.138.141.172 box.
You cut off your config - are you trying to do auth?
If you want to add more later - sure but for troubleshooting I would just point to your 1 local server.. And then sniff do you see an answer?
Can we see your other devices that are syncing off this local server? For example I already showed my windows box syncing - here is one of my switches using it
sg300-28#sho sntp status Clock is synchronized, stratum 1, reference is 192.168.3.32, unicast Unicast servers: Server : 192.168.3.32 Source : Static Stratum : 1 Status : up Last Response : 17:03:25.0 web Jun 2 2024 Offset : 347.3516068 mSec Delay : -1000 mSec
here is my nas, and printer all pointing to my local ntp - showing working
For troubleshooting - I would set the only interface on to be the one interface where your ntp server is. And I would sniff the traffic to validate you see it sending to it, and getting a response on its own accord, not with the ntpdate command, etc.
Here is my unifi controller setting, and showing that one of the AP is sync'd etc
edit: oh my nas is using old name, which still resolves - but should change that to home.arpa vs local.lan
What are you running for your ntp server? ntp, ntpsec, chrony?
edit2: that you are only able to talk to 1 of the many many ntp servers your pointing to is very odd to me.. If there was something wrong with pfsense, why is talking to anything. If there was some disconnect between pfsense and your ntp server, why is it not talking to all the other servers your pointing to?
Clearly something is amiss - figuring out what it is the puzzle needed to be solved.
Appreciate the reply.
I'm not doing NTPv3 authentication.
I actually originally had it only pointing to my local server, but when it started acting up, I started adding. I assumed that since I checked "prefer" it would default to that one if it started working.
Here's what I get when I delete all the others.
I can point anything on the network to my NTP and it seems to work. Here's the windows 10 machine I'm on now.
Here's a TrueNAS server that I just pointed it to.
Really, I only need it for my Blue Iris machine because I want all of my security cameras synced. Other than that, it's just nice to have it working.
Forgive me, but I'm not sure how do do the ole sniff.
-
@ARAMP1 the sniff would be just packet capture under the diagnostic menu.. You can have it only capture the ntp from your IP of your ntp server and port 123.
Select the interface used to talk to your ntp server, assuming that is your lan interface.
You can then download that capture and we can look at it with say wireshark.
The info from your nas is good.. what IP is your nas on, I assume its on this same 192.168.10 network.
edit: don't camera's sync their time via ipc?
I don't have ntp setup on my cameras - they sync with the nvr via ipc.. But blue iris could be different for sure. The nvr syncs directly with my ntp server.. Why wouldn't you sync your blue iris and your cameras directly with your ntp server? vs pfsense?
That being said - pfsense for sure should be able to sync with your ntp server.. But if you can just sync your camera stuff with your ntp server, then the pfsense time sync isn't as critical, etc. And can figure out what is going on it with it without having to worry about your camera time being off, etc.
-
@ARAMP1 the sniff would be just packet capture under the diagnostic menu.. You can have it only capture the ntp from your IP of your ntp server and port 123.
Select the interface used to talk to your ntp server, assuming that is your lan interface.
You can then download that capture and we can look at it with say wireshark.
The info from your nas is good.. what IP is your nas on, I assume its on this same 192.168.10 network.
My windows machine is on the 10.10.50 network and that truenas machine is on the 192.168.60 network but I have rules on each of those VLANs to access 192.168.10.120 for NTP (the 192.168.10 network is my management network with all my switches, access points, etc).
For cameras, here's the rules I have in place.
-
@ARAMP1 well that is interesting.. So the devices on different networks sync fine - but the pfsense on the same network is not able to reach it??
Your not doing anything with ethernet filtering, nor any floating rules for outbound filtering?
As a test - your truenas can act as ntp server I would assume.. Can you point pfsense to it?
Example - just pointed pfsense to my nas.
notice it shows the 9.10 address of my nas, and it shows its ref as my pi ntp server the 3.32 IP
-
Here's a short run of the packet capture...
00:56:11.413566 IP 192.168.60.251.123 > 192.168.10.120.123: UDP, length 48
00:56:11.413612 IP 192.168.60.251.123 > 192.168.10.120.123: UDP, length 48
00:56:11.414620 IP 192.168.10.120.123 > 192.168.60.251.123: UDP, length 48
00:56:11.414652 IP 192.168.10.120.123 > 192.168.60.251.123: UDP, length 48
00:57:17.413522 IP 192.168.60.251.123 > 192.168.10.120.123: UDP, length 48
00:57:17.413531 IP 192.168.60.251.123 > 192.168.10.120.123: UDP, length 48
00:57:17.414624 IP 192.168.10.120.123 > 192.168.60.251.123: UDP, length 48
00:57:17.414659 IP 192.168.10.120.123 > 192.168.60.251.123: UDP, length 48
00:58:23.373022 IP 192.168.60.251.123 > 192.168.10.120.123: UDP, length 48
00:58:23.373061 IP 192.168.60.251.123 > 192.168.10.120.123: UDP, length 48
00:58:23.373684 IP 192.168.10.120.123 > 192.168.60.251.123: UDP, length 48
00:58:23.373718 IP 192.168.10.120.123 > 192.168.60.251.123: UDP, length 48
00:58:42.477178 IP 10.10.50.230.44863 > 192.168.10.120.123: UDP, length 48
00:58:42.477281 IP 10.10.50.230.44863 > 192.168.10.120.123: UDP, length 48
00:58:42.477576 IP 192.168.10.120.123 > 10.10.50.230.44863: UDP, length 48
00:58:42.477597 IP 192.168.10.120.123 > 10.10.50.230.44863: UDP, length 48
00:58:43.586214 IP 10.10.50.230.50913 > 192.168.10.120.123: UDP, length 48
00:58:43.586317 IP 10.10.50.230.50913 > 192.168.10.120.123: UDP, length 48
00:58:43.586555 IP 192.168.10.120.123 > 10.10.50.230.50913: UDP, length 48
00:58:43.586560 IP 192.168.10.120.123 > 10.10.50.230.50913: UDP, length 48
00:58:45.604892 IP 10.10.50.230.41040 > 192.168.10.120.123: UDP, length 48
00:58:45.605003 IP 10.10.50.230.41040 > 192.168.10.120.123: UDP, length 48
00:58:45.605557 IP 192.168.10.120.123 > 10.10.50.230.41040: UDP, length 48
00:58:45.605580 IP 192.168.10.120.123 > 10.10.50.230.41040: UDP, length 48
00:58:47.636203 IP 10.10.50.230.54265 > 192.168.10.120.123: UDP, length 48
00:58:47.636249 IP 10.10.50.230.54265 > 192.168.10.120.123: UDP, length 48
00:58:47.636552 IP 192.168.10.120.123 > 10.10.50.230.54265: UDP, length 48
00:58:47.636578 IP 192.168.10.120.123 > 10.10.50.230.54265: UDP, length 48
00:58:49.778857 IP 10.10.50.230.56448 > 192.168.10.120.123: UDP, length 48
00:58:49.778967 IP 10.10.50.230.56448 > 192.168.10.120.123: UDP, length 48
00:58:49.779532 IP 192.168.10.120.123 > 10.10.50.230.56448: UDP, length 48
00:58:49.779552 IP 192.168.10.120.123 > 10.10.50.230.56448: UDP, length 48
00:58:51.786147 IP 10.10.50.230.36563 > 192.168.10.120.123: UDP, length 48
00:58:51.786273 IP 10.10.50.230.36563 > 192.168.10.120.123: UDP, length 48
00:58:51.786525 IP 192.168.10.120.123 > 10.10.50.230.36563: UDP, length 48
00:58:51.786529 IP 192.168.10.120.123 > 10.10.50.230.36563: UDP, length 48
00:58:53.816726 IP 10.10.50.230.58706 > 192.168.10.120.123: UDP, length 48
00:58:53.816798 IP 10.10.50.230.58706 > 192.168.10.120.123: UDP, length 48
00:58:53.817676 IP 192.168.10.120.123 > 10.10.50.230.58706: UDP, length 48
00:58:53.817681 IP 192.168.10.120.123 > 10.10.50.230.58706: UDP, length 48
00:58:54.780841 IP 10.10.50.230.56213 > 192.168.10.120.123: UDP, length 48
00:58:54.780910 IP 10.10.50.230.56213 > 192.168.10.120.123: UDP, length 48
00:58:54.781525 IP 192.168.10.120.123 > 10.10.50.230.56213: UDP, length 48
00:58:54.781533 IP 192.168.10.120.123 > 10.10.50.230.56213: UDP, length 48
00:58:56.791962 IP 10.10.50.230.46158 > 192.168.10.120.123: UDP, length 48
00:58:56.792080 IP 10.10.50.230.46158 > 192.168.10.120.123: UDP, length 48
00:58:56.792530 IP 192.168.10.120.123 > 10.10.50.230.46158: UDP, length 48
00:58:56.792551 IP 192.168.10.120.123 > 10.10.50.230.46158: UDP, length 48
00:58:58.467266 IP 10.10.50.230.54383 > 192.168.10.120.123: UDP, length 48
00:58:58.467305 IP 10.10.50.230.54383 > 192.168.10.120.123: UDP, length 48
00:58:58.467569 IP 192.168.10.120.123 > 10.10.50.230.54383: UDP, length 48
00:58:58.467574 IP 192.168.10.120.123 > 10.10.50.230.54383: UDP, length 48
00:59:00.508052 IP 10.10.50.230.44608 > 192.168.10.120.123: UDP, length 48
00:59:00.508113 IP 10.10.50.230.44608 > 192.168.10.120.123: UDP, length 48
00:59:00.508564 IP 192.168.10.120.123 > 10.10.50.230.44608: UDP, length 48
00:59:00.508568 IP 192.168.10.120.123 > 10.10.50.230.44608: UDP, length 48
00:59:02.576259 IP 10.10.50.230.54479 > 192.168.10.120.123: UDP, length 48
00:59:02.576371 IP 10.10.50.230.54479 > 192.168.10.120.123: UDP, length 48
00:59:02.576552 IP 192.168.10.120.123 > 10.10.50.230.54479: UDP, length 48
00:59:02.576558 IP 192.168.10.120.123 > 10.10.50.230.54479: UDP, length 48
00:59:04.587321 IP 10.10.50.230.34257 > 192.168.10.120.123: UDP, length 48
00:59:04.587386 IP 10.10.50.230.34257 > 192.168.10.120.123: UDP, length 48
00:59:04.587550 IP 192.168.10.120.123 > 10.10.50.230.34257: UDP, length 48
00:59:04.587556 IP 192.168.10.120.123 > 10.10.50.230.34257: UDP, length 48
00:59:06.612532 IP 10.10.50.230.37136 > 192.168.10.120.123: UDP, length 48
00:59:06.612592 IP 10.10.50.230.37136 > 192.168.10.120.123: UDP, length 48
00:59:06.613552 IP 192.168.10.120.123 > 10.10.50.230.37136: UDP, length 48
00:59:06.613578 IP 192.168.10.120.123 > 10.10.50.230.37136: UDP, length 48
00:59:08.641801 IP 10.10.50.230.33174 > 192.168.10.120.123: UDP, length 48
00:59:08.641915 IP 10.10.50.230.33174 > 192.168.10.120.123: UDP, length 48
00:59:08.642548 IP 192.168.10.120.123 > 10.10.50.230.33174: UDP, length 48
00:59:08.642570 IP 192.168.10.120.123 > 10.10.50.230.33174: UDP, length 48
00:59:29.381935 IP 192.168.60.251.123 > 192.168.10.120.123: UDP, length 48
00:59:29.381971 IP 192.168.60.251.123 > 192.168.10.120.123: UDP, length 48
00:59:29.382608 IP 192.168.10.120.123 > 192.168.60.251.123: UDP, length 48
00:59:29.382641 IP 192.168.10.120.123 > 192.168.60.251.123: UDP, length 48
01:00:13.721904 IP 10.10.50.230.48115 > 192.168.10.120.123: UDP, length 48
01:00:13.722019 IP 10.10.50.230.48115 > 192.168.10.120.123: UDP, length 48
01:00:13.722520 IP 192.168.10.120.123 > 10.10.50.230.48115: UDP, length 48
01:00:13.722524 IP 192.168.10.120.123 > 10.10.50.230.48115: UDP, length 48
01:00:38.413534 IP 192.168.60.251.123 > 192.168.10.120.123: UDP, length 48
01:00:38.413568 IP 192.168.60.251.123 > 192.168.10.120.123: UDP, length 48
01:00:38.414569 IP 192.168.10.120.123 > 192.168.60.251.123: UDP, length 48
01:00:38.414598 IP 192.168.10.120.123 > 192.168.60.251.123: UDP, length 48
01:00:52.473322 IP 192.168.10.1.123 > 192.168.10.120.123: UDP, length 48
01:00:52.473549 IP 192.168.10.120.123 > 192.168.10.1.123: UDP, length 48
01:00:55.450351 IP 192.168.10.1.123 > 192.168.10.120.123: UDP, length 48
01:00:55.450552 IP 192.168.10.120.123 > 192.168.10.1.123: UDP, length 48
01:01:00.525365 IP 192.168.10.1.123 > 192.168.10.120.123: UDP, length 48
01:01:00.525543 IP 192.168.10.120.123 > 192.168.10.1.123: UDP, length 48
01:01:18.317945 IP 10.10.50.230.38268 > 192.168.10.120.123: UDP, length 48
01:01:18.318057 IP 10.10.50.230.38268 > 192.168.10.120.123: UDP, length 48
01:01:18.318613 IP 192.168.10.120.123 > 10.10.50.230.38268: UDP, length 48
01:01:18.318635 IP 192.168.10.120.123 > 10.10.50.230.38268: UDP, length 48
01:01:47.398970 IP 192.168.60.251.123 > 192.168.10.120.123: UDP, length 48
01:01:47.399009 IP 192.168.60.251.123 > 192.168.10.120.123: UDP, length 48
01:01:47.399617 IP 192.168.10.120.123 > 192.168.60.251.123: UDP, length 48
01:01:47.399650 IP 192.168.10.120.123 > 192.168.60.251.123: UDP, length 48
01:02:05.525345 IP 192.168.10.1.123 > 192.168.10.120.123: UDP, length 48
01:02:05.525544 IP 192.168.10.120.123 > 192.168.10.1.123: UDP, length 48
01:02:22.690101 IP 10.10.50.230.44520 > 192.168.10.120.123: UDP, length 48
01:02:22.690213 IP 10.10.50.230.44520 > 192.168.10.120.123: UDP, length 48
01:02:22.690525 IP 192.168.10.120.123 > 10.10.50.230.44520: UDP, length 48
01:02:22.690546 IP 192.168.10.120.123 > 10.10.50.230.44520: UDP, length 48
01:02:54.378938 IP 192.168.60.251.123 > 192.168.10.120.123: UDP, length 48
01:02:54.378974 IP 192.168.60.251.123 > 192.168.10.120.123: UDP, length 48
01:02:54.379592 IP 192.168.10.120.123 > 192.168.60.251.123: UDP, length 48
01:02:54.379623 IP 192.168.10.120.123 > 192.168.60.251.123: UDP, length 48
01:03:09.525230 IP 192.168.10.1.123 > 192.168.10.120.123: UDP, length 48
01:03:09.525545 IP 192.168.10.120.123 > 192.168.10.1.123: UDP, length 48
01:03:27.350112 IP 10.10.50.230.52709 > 192.168.10.120.123: UDP, length 48
01:03:27.350221 IP 10.10.50.230.52709 > 192.168.10.120.123: UDP, length 48
01:03:27.350617 IP 192.168.10.120.123 > 10.10.50.230.52709: UDP, length 48
01:03:27.350637 IP 192.168.10.120.123 > 10.10.50.230.52709: UDP, length 48 -
01:02:05.525345 IP 192.168.10.1.123 > 192.168.10.120.123: UDP, length 48
01:02:05.525544 IP 192.168.10.120.123 > 192.168.10.1.123: UDP, length 48I take it that is your pfsense IP at 10.1 - can you download that and attach the pcap.. For example - here is pcap I did earlier
-
@ARAMP1 well that is interesting.. So the devices on different networks sync fine - but the pfsense on the same network is not able to reach it??
Your not doing anything with ethernet filtering, nor any floating rules for outbound filtering?
As a test - your truenas can act as ntp server I would assume.. Can you point pfsense to it?
Example - just pointed pfsense to my nas.
notice it shows the 9.10 address of my nas, and it shows its ref as my pi ntp server the 3.32 IP
Just tried pointing it to a couple truenas servers I have on different VLANs. Both are Unreach/Pending.
The floating rules I have in place only have to do with CoDel traffic limiter and pfBlocker.
-
-
@ARAMP1 this seems a bit odd
-
@johnpoz I'm looking at wireshark now, but I don't know what I'm looking at.
-
-
@ARAMP1 no it wouldn't have anything to do with dns, because your pointing to an IP for the server.
I would think that ref ID would point to the IP your asking.. see in mine it shows my 192.168.3.32?
It might be like that because it has never gotten an answer? So it has no ref as of yet?
From your sniff you get an answer, so why would ntp think it can not reach that server?
-
@ARAMP1 can you turn off ntp and validate nothing is listening on 123..
example
-
packetcapture-ix1-20240602204723.pcap
I've still pointed clients at 192.168.10.120 directly, so that would still show something listening on port 123, right?
-
@ARAMP1 not clicked to not enable it
As you see when I disable it nothing is listening on 123
Also can you one you enable it again, post up your ntpd.conf
[23.09.1-RELEASE][admin@sg4860.home.arpa]/: cat /var/etc/ntpd.conf # # pfSense ntp configuration file # tinker panic 0 # Orphan mode stratum and Maximum candidate NTP peers tos orphan 12 maxclock 5 # Upstream Servers server -4 192.168.3.32 iburst minpoll 6 maxpoll 10 prefer enable stats statistics clockstats loopstats peerstats statsdir /var/log/ntp logconfig =syncall +clockall +peerall +sysall driftfile /var/db/ntpd.drift restrict default kod limited nomodify nopeer notrap restrict -6 default kod limited nomodify nopeer notrap interface ignore all interface ignore wildcard interface listen igb2.1011 interface listen igb3 interface listen igb0 interface listen igb4 interface listen igb2 interface listen igb2.110 interface listen igb2.6 interface listen igb2.4 interface listen igb5 interface listen lo0 [23.09.1-RELEASE][admin@sg4860.home.arpa]/:
-
-
@ARAMP1 ok the only thing that looks odd is your restrict 192.168.10.120 that is not really a network that is a host address. The network with /24 would be 192.168.10.0
After seeing yours noticed mine wasn't even listed... I removed mine completely because to be honest that is for clients wanting to talk to pfsense as their server. So you shouldn't even need that.
I would remove it, but that would be really crazy if that was the issue.
Nothing is there that makes any sense to why it wouldn't be working that I am seeing..
So you have no native network on the ix1 interface.. Only vlans.. I take it ix1.10 is your 192.168.10 interface..
-
@johnpoz I'm not following when you say "I would remove it". What would/did you remove?
I have a native network on the ix1 interface but it's not enabled.