NTP and stratum issue

fatsailor · Jan 1, 2016, 10:04 PM

I'm trying to use pfsense (2.2.6) as a time server for the LAN.

Clients are failing because the stratum being returned is 16 when I debug with 'ntpdate -d pfsense':

stratum 16, precision -6, leap 11, trust 000

When I use ntpq on the pfsense box - the 'rv' command tells me the stratum is 2.

ntpq> rv
associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync,
version="ntpd 4.2.8p4@1.3265-o Mon Oct 26 14:28:17 UTC 2015 (1)",
processor="amd64", system="FreeBSD/10.1-RELEASE-p25", leap=00, stratum=2,
precision=-23, rootdelay=63.249, rootdisp=10.466, refid=132.163.4.101,
reftime=da3174e2.f45f84b6 Fri, Jan 1 2016 16:57:54.954,
clock=da3175dc.165390e2 Fri, Jan 1 2016 17:02:04.087, peer=16120, tc=7,
mintc=3, offset=2.566951, frequency=18.819, sys_jitter=2.789931,
clk_jitter=4.064, clk_wander=0.035

I'm pretty confused on how to fix this. I'd like the firewall to be the time sever, but I don't know how to debug this any further.

For completeness, here's the output of peers, and as.

ntpq> peers
remote refid st t when poll reach delay offset jitter

*time-a.timefreq .ACTS. 1 u 122 128 377 63.249 2.567 2.406
+host-24-56-178- .ACTS. 1 u 51 128 377 55.824 12.147 0.818
+gps.layer42.net .GPS. 1 u 59 128 377 72.650 15.905 3.152
-time-b.nist.gov .ACTS. 1 u 57 128 173 70.204 47.305 1.220

ntpq> as

ind assid status conf reach auth condition last_event cnt

1 16120 963a yes yes none sys.peer sys_peer 3
2 16121 944d yes yes none candidate 4
3 16122 9424 yes yes none candidate reachable 2
4 16123 9324 yes yes none outlier reachable 2

johnpoz · Jan 1, 2016, 10:19 PM

You also got a Server dropped: Leap not in sync in your ntpdate did you not… I see your leap bit set to 11 there.. But your rv doesn't show that.. You sure your not just resolving pfsense to the wrong time server?

fatsailor · Jan 1, 2016, 10:50 PM

I'm positive I'm hitting the right IP for the pfsense box.

I'll delete the unresponsive time server, and see what that does for us.

fatsailor · Jan 1, 2016, 11:08 PM

More info - if I do a ntpdate -ud localhost, I get results that match ntpq rv. If I run the same command from any machine on the LAN, I get stratum 16.

NOTE: I also shut the NTP service down on the pfsense box, and I confirmed that I could no longer reach the service from the clients and localhost.

10.1.1.1 is the pfsense box

$ ssh admin@10.1.1.1
*** Welcome to pfSense 2.2.6-RELEASE-pfSense (amd64) on pfSense ***

Running against localhost

[2.2.6-RELEASE][admin@pfSense]/root: ntpdate -ud localhost
1 Jan 17:54:08 ntpdate[13292]: ntpdate 4.2.8p4@1.3265-o Mon Oct 26 14:28:19 UTC 2015 (1)
Looking for host localhost and service ntp
127.0.0.1 reversed to localhost
host found : localhost
transmit(127.0.0.1)
receive(127.0.0.1)
transmit(::1)
receive(::1)
transmit(127.0.0.1)
receive(127.0.0.1)
transmit(::1)
receive(::1)
transmit(127.0.0.1)
receive(127.0.0.1)
transmit(::1)
receive(::1)
transmit(127.0.0.1)
receive(127.0.0.1)
transmit(::1)
receive(::1)
server 127.0.0.1, port 123
stratum 3, precision -23, leap 00, trust 000
refid [127.0.0.1], delay 0.02567, dispersion 0.00000
transmitted 4, in filter 4
reference time: da3181e4.9049c7d3 Fri, Jan 1 2016 17:53:24.563
originate timestamp: da318216.52f855dc Fri, Jan 1 2016 17:54:14.324
transmit timestamp: da318216.52f477d5 Fri, Jan 1 2016 17:54:14.324
filter delay: 0.02573 0.02568 0.02567 0.02568
0.00000 0.00000 0.00000 0.00000
filter offset: 0.000024 -0.00000 -0.00000 -0.00000
0.000000 0.000000 0.000000 0.000000
delay 0.02567, dispersion 0.00000
offset -0.000002

server ::1, port 123
stratum 3, precision -23, leap 00, trust 000
refid [::1], delay 0.02568, dispersion 0.00000
transmitted 4, in filter 4
reference time: da3181e4.9049c7d3 Fri, Jan 1 2016 17:53:24.563
originate timestamp: da318216.862b731b Fri, Jan 1 2016 17:54:14.524
transmit timestamp: da318216.8627c7d6 Fri, Jan 1 2016 17:54:14.524
filter delay: 0.02573 0.02570 0.02568 0.02570
0.00000 0.00000 0.00000 0.00000
filter offset: 0.000018 0.000004 0.000003 0.000004
0.000000 0.000000 0.000000 0.000000
delay 0.02568, dispersion 0.00000
offset 0.000003

1 Jan 17:54:14 ntpdate[13292]: adjust time server 127.0.0.1 offset -0.000002 sec

Running from a client

[root@test] ~# ntpdate -ud 10.1.1.1
1 Jan 17:54:52 ntpdate[8615]: ntpdate 4.2.4p5-a (1)
transmit(10.1.1.1)
receive(10.1.1.1)
transmit(10.1.1.1)
receive(10.1.1.1)
transmit(10.1.1.1)
transmit(10.1.1.1)
transmit(10.1.1.1)
10.1.1.1: Server dropped: strata too high
server 10.1.1.1, port 123
stratum 16, precision -6, leap 11, trust 000
refid [10.1.1.1], delay 0.04140, dispersion 24.07111
transmitted 4, in filter 4
reference time: 00000000.00000000 Thu, Feb 7 2036 1:28:16.000
originate timestamp: da31823c.d7c0ee0f Fri, Jan 1 2016 17:54:52.842
transmit timestamp: da31823d.ba086f2d Fri, Jan 1 2016 17:54:53.726
filter delay: 0.14259 0.04140 0.00000 0.00000
0.00000 0.00000 0.00000 0.00000
filter offset: 0.142123 -0.00008 0.000000 0.000000
0.000000 0.000000 0.000000 0.000000
delay 0.04140, dispersion 24.07111
offset -0.000080

1 Jan 17:54:54 ntpdate[8615]: no server suitable for synchronization found

johnpoz · Jan 2, 2016, 1:13 AM

Dude your not hitting the same server it looks to me.. You forwarding traffic?

On pfsense change your host to your pfsense IP and do rv, and do same thing from localhost

They should be exactly the same

ntpq> host localhost
current host set to localhost
ntpq> rv
associd=0 status=0618 leap_none, sync_ntp, 1 event, no_sys_peer,
version="ntpd 4.2.8p4@1.3265-o Mon Oct 26 14:28:17 UTC 2015 (1)",
processor="amd64", system="FreeBSD/10.1-RELEASE-p25", leap=00, stratum=3,
precision=-18, rootdelay=13.319, rootdisp=59.874, refid=192.168.9.40,
reftime=da319e93.4960f3d0 Fri, Jan 1 2016 18:55:47.286,
clock=da31a29e.32685bb3 Fri, Jan 1 2016 19:13:02.196, peer=20504, tc=8,
mintc=3, offset=0.475810, frequency=-55.601, sys_jitter=0.230866,
clk_jitter=0.087, clk_wander=0.002
ntpq> host 192.168.9.253
current host set to 192.168.9.253
ntpq> rv
associd=0 status=0618 leap_none, sync_ntp, 1 event, no_sys_peer,
version="ntpd 4.2.8p4@1.3265-o Mon Oct 26 14:28:17 UTC 2015 (1)",
processor="amd64", system="FreeBSD/10.1-RELEASE-p25", leap=00, stratum=3,
precision=-18, rootdelay=13.319, rootdisp=60.129, refid=192.168.9.40,
reftime=da319e93.4960f3d0 Fri, Jan 1 2016 18:55:47.286,
clock=da31a2ae.87be472a Fri, Jan 1 2016 19:13:18.530, peer=20504, tc=8,
mintc=3, offset=0.475810, frequency=-55.601, sys_jitter=0.230866,
clk_jitter=0.087, clk_wander=0.002
ntpq>

fatsailor · Jan 2, 2016, 1:37 AM

ummm - yeah I am. The weird thing is that 'ntpdate -ud 10.1.1.1' and 'ntpq -c rv 10.1.1.1' return different results on a client box when pointed to the pfsense box.

I know ntpdate is being depreciated but this is just a bit nuts.

I've clients, e.g. the Freenas box, that won't accept the pfsense box as a time server because of the stratum issue. I'm wondering if this is a software version mismatch issue?

This is all done on the pfsense box

ntpq> host localhost
current host set to localhost
ntpq> rv
associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync,
version="ntpd 4.2.8p4@1.3265-o Mon Oct 26 14:28:17 UTC 2015 (1)",
processor="amd64", system="FreeBSD/10.1-RELEASE-p25", leap=00, stratum=2,
precision=-23, rootdelay=24.781, rootdisp=585.121, refid=204.9.54.119,
reftime=da31a54c.232af4c7 Fri, Jan 1 2016 20:24:28.137,
clock=da31a561.e76f2bd6 Fri, Jan 1 2016 20:24:49.904, peer=58444, tc=6,
mintc=3, offset=28.685194, frequency=18.604, sys_jitter=117.371580,
clk_jitter=7.981, clk_wander=0.000
ntpq> host 10.1.1.1
current host set to 10.1.1.1
ntpq> rv
associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync,
version="ntpd 4.2.8p4@1.3265-o Mon Oct 26 14:28:17 UTC 2015 (1)",
processor="amd64", system="FreeBSD/10.1-RELEASE-p25", leap=00, stratum=2,
precision=-23, rootdelay=24.781, rootdisp=585.436, refid=204.9.54.119,
reftime=da31a54c.232af4c7 Fri, Jan 1 2016 20:24:28.137,
clock=da31a576.73554a5e Fri, Jan 1 2016 20:25:10.450, peer=58444, tc=6,
mintc=3, offset=28.685194, frequency=18.604, sys_jitter=117.371580,
clk_jitter=7.981, clk_wander=0.000
ntpq>

This is done on a Freenas box

ntpq> host 10.1.1.1
current host set to 10.1.1.1
ntpq> rv
assID=0 status=0615 leap_none, sync_ntp, 1 event, event_clock_reset,
version="ntpd 4.2.8p4@1.3265-o Mon Oct 26 14:28:17 UTC 2015 (1)",
processor="amd64", system="FreeBSD/10.1-RELEASE-p25", leap=00,
stratum=2, precision=-23, rootdelay=25.191, rootdisp=67.854,
refid=204.9.54.119,
reftime=da31a5c9.235647a3 Fri, Jan 1 2016 20:26:33.138,
clock=da31a622.b3b0f7cd Fri, Jan 1 2016 20:28:02.701, peer=58444,
tc=6, mintc=3, offset=-0.122, frequency=18.603, sys_jitter=22.813403,
clk_jitter=12.192, clk_wander=0.000
ntpq> q
[root@freenas] ~# ntpdate -ud 10.1.1.1
1 Jan 20:28:46 ntpdate[15064]: ntpdate 4.2.4p5-a (1)
transmit(10.1.1.1)
receive(10.1.1.1)
transmit(10.1.1.1)
receive(10.1.1.1)
transmit(10.1.1.1)
transmit(10.1.1.1)
transmit(10.1.1.1)
10.1.1.1: Server dropped: strata too high
server 10.1.1.1, port 123
stratum 16, precision -6, leap 11, trust 000
refid [10.1.1.1], delay 0.02571, dispersion 24.01199
transmitted 4, in filter 4
reference time: 00000000.00000000 Thu, Feb 7 2036 1:28:16.000
originate timestamp: da31a64e.39fe35ff Fri, Jan 1 2016 20:28:46.226
transmit timestamp: da31a64f.3a2ea971 Fri, Jan 1 2016 20:28:47.227
filter delay: 0.02571 0.04134 0.00000 0.00000
0.00000 0.00000 0.00000 0.00000
filter offset: 0.023948 -0.00004 0.000000 0.000000
0.000000 0.000000 0.000000 0.000000
delay 0.02571, dispersion 24.01199
offset 0.023948

1 Jan 20:28:48 ntpdate[15064]: no server suitable for synchronization found
[root@freenas] ~#

johnpoz · Jan 2, 2016, 3:48 AM

you are using an older version of ntpdate.. Why don't you just update it?

Current dev version is 4.3.88 it is syncing fine to pfsense ntp server

user@clean:~$ ntpdate -d pfsense.local.lan
1 Jan 21:46:42 ntpdate[29545]: ntpdate 4.3.88@1.2483 Mon Dec 28 22:23:20 UTC 2015 (1)
Looking for host pfsense.local.lan and service ntp
192.168.9.253 reversed to pfSense.local.lan
host found : pfSense.local.lan
transmit(192.168.9.253)
receive(192.168.9.253)
transmit(192.168.9.253)
receive(192.168.9.253)
transmit(192.168.9.253)
receive(192.168.9.253)
transmit(192.168.9.253)
receive(192.168.9.253)
server 192.168.9.253, port 123
stratum 3, precision -18, leap 00, trust 000
refid [192.168.9.253], delay 0.02600, dispersion 0.00003
transmitted 4, in filter 4
reference time: da31c5f2.447ed54e Fri, Jan 1 2016 21:43:46.267
originate timestamp: da31c6a8.e20bc1c1 Fri, Jan 1 2016 21:46:48.882
transmit timestamp: da31c6a8.e1e17584 Fri, Jan 1 2016 21:46:48.882
filter delay: 0.02606 0.02605 0.02606 0.02600
0.00000 0.00000 0.00000 0.00000
filter offset: 0.000203 0.000183 0.000202 0.000250
0.000000 0.000000 0.000000 0.000000
delay 0.02600, dispersion 0.00003
offset 0.000250

1 Jan 21:46:48 ntpdate[29545]: adjust time server 192.168.9.253 offset 0.000250 sec
user@clean:~$

fatsailor · Jan 2, 2016, 1:19 PM

Thanks for the help! I love being the first to find version mismatch bugs.

Rather than update the entire office of OSX clients and Freenas servers to a newer version of ntpdate, I think we're just going to make ntpdate an alias to 'ntpd -qg -c <special config="">' or something along those lines. That way we don't have to worry about dependencies etc.

I'll probably file a bug with Freenas also so someone else doesn't waste an afternoon chasing this one down.</special>

johnpoz · Jan 2, 2016, 2:06 PM

Dude but isn't 4.2.4p5 really really freaking old?? didn't 4.2.4p5 come out in like Aug of 2008?? What version of os x and freebsd are you running that that would be the ntp version?

So help validate this, I grabbed an OLD version for windows - couldn't find the exact 4.2.4p5 but p8 pretty close, and yeah fails..

slu · Jan 3, 2016, 12:24 PM

@fatsailor:

stratum 16, precision -6, leap 11, trust 000

See exactly the same with a older CentOS (guess 6.4) Server, Debian 7/8 works.

johnpoz · Jan 3, 2016, 12:26 PM

Yeah it looks like there is a issue between versions of ntp.. What version of ntp is the centos box running

slu · Jan 3, 2016, 12:44 PM

@johnpoz:

What version of ntp is the centos box running

Don't know which version was affected, now it is a CentOS 6.7 with ntp.x86_64 4.2.6p5-5.el6.centos.2 and it working as expected.

David_W · Jan 3, 2016, 12:45 PM

ntpdate is rather bitrotted and has been on the pathway to deprecation for some time.

Depending on the usage case, sntp or ntpd -gq should be used - the link in the previous paragraph gives some background and implementation considerations.

FreeBSD base system only abandoned ntp 4.2.4 in the second half of 2015.

charliem · Jan 4, 2016, 10:06 PM

@fatsailor:

Rather than update the entire office of OSX clients and Freenas servers to a newer version of ntpdate, I think we're just going to make ntpdate an alias to 'ntpd -qg -c <special config="">' or something along those lines. That way we don't have to worry about dependencies etc.

I'll probably file a bug with Freenas also so someone else doesn't waste an afternoon chasing this one down.</special>

It was reported here that choosing certain parameters in NTP access restrictions would work around this bug in older ntpdate client versions. I haven't tested myself, but if it works then you wouldn't have to touch each client.

Yes, 'ntpd -gq' is the right approach going forward, but it may not be practical to upgrade client machines.

johnpoz · Jan 5, 2016, 9:53 AM

So looking at that thread.. And a simple test with old ntpdate, its the KOD packet that is killing the old version for some reason..

If you uncheck the Kiss-o'-death, then ntpdate from old client works… If you have it enabled then it fails with stratum 16, and leap 11

edit: Seems if kod is enabled, it also places a limited in the restrictions, which is a conflict when you have no monitor set as well, which is in the conf file but don't see anyway in the gui to edit that.

Jan 5 04:05:38 ntpd[11014]: restrict: 'monitor' cannot be disabled while 'limited' is enabled

If you uncheck kod, then the kod is removed from the conf as well as the limited.

disable monitor
statsdir /var/log/ntp
logconfig =syncall +clockall +peerall +sysall
driftfile /var/db/ntpd.drift
restrict default nomodify nopeer notrap
restrict -6 default nomodify nopeer notrap

If you check the kod option in access restrictions you get
disable monitor
statsdir /var/log/ntp
logconfig =syncall +clockall +peerall +sysall
driftfile /var/db/ntpd.drift
restrict default kod limited nomodify nopeer notrap
restrict -6 default kod limited nomodify nopeer notrap

David_W · Jan 8, 2016, 7:36 PM

@johnpoz:

And a simple test with old ntpdate, its the KOD packet that is killing the old version for some reason..

If you uncheck the Kiss-o'-death, then ntpdate from old client works… If you have it enabled then it fails with stratum 16, and leap 11

In this case, kiss-of-death is working as intended, but ntpdate earlier that some time during the 4.2.7 cycle fails to understand it.

Dave Hart posted an excellent analysis of this scenario in the comp.protocols.time.ntp newsgroup.

I found:
discard average 6 minimum 0
got my ntp servers working with old ntpdate binaries and kiss-of-death enabled.

johnpoz · Jan 9, 2016, 2:45 AM

why wouldn't you update vs downgrade.. I show it working with current ntpdate.

David_W · Jan 9, 2016, 2:52 AM

@johnpoz:

why wouldn't you update vs downgrade.. I show it working with current ntpdate.

It's a compatibility thing. Nothing here runs ntp older than 4.2.8p4 (4.2.8p5 has just released), but there are circumstances where older remote clients need to run against my servers.

HeMaN · Mar 7, 2017, 10:32 AM

Thank you for this topic. Was pulling my hair out why my Thecus nas was the only device not able to sync with time my pfsense box.
All test done from my windows pc showed no issues. Looking on the nas I found out it is not using ntpd / ntpq but ntpdate (ntpdate 4.2.4p8 to be exact)

throug a long way of searching came to this topic.

After setting "ACL - Custom Access Restrictions" in pfSense for the IP of my NAS and disabeling KOD, all is working fine

johnpoz · Mar 7, 2017, 11:58 AM

Glad the topic was of use of you HeMan.. When I saw this show up as something new it - was at first ah shit some spammer necro an old thread ;) Nice to see it was someone actual used the info contained in old threads for what they are meant for ;)

NTP and stratum issue

ntpq> peers remote refid st t when poll reach delay offset jitter

ind assid status conf reach auth condition last_event cnt

ntpq> peers
remote refid st t when poll reach delay offset jitter