NTP - poor reach after 20 hours

farmerjohn · Apr 15, 2021, 11:11 PM

@johnpoz
i updated from 2.5.0 to 2.5.1. Here is NTP status after 90 minutes after reboot. Hopefully the Scandinavian guys won't try and shame me again with their impressive NTP output.

[2.5.1-RELEASE][admin@pfsense.home]/var/log: ntpq -pcrv
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.pfsense.pool. .POOL.          16 p    -   64    0    0.000   +0.000   0.000
+unifi.versadns. 71.66.197.233    2 u  632  128  120   42.354   -2.973   1.580
*clock.nyc.he.ne .CDMA.           1 u  453  128  144   37.191   +1.439   1.048

associd=0 status=0618 leap_none, sync_ntp, 1 event, no_sys_peer,
version="ntpd 4.2.8p15@1.3728-o Fri Feb  5 22:07:56 UTC 2021 (1)",
processor="amd64", system="FreeBSD/12.2-STABLE", leap=00, stratum=2,
precision=-21, rootdelay=37.191, rootdisp=476.256, refid=209.51.161.238,
reftime=e4233dba.ae57b7eb  Thu, Apr 15 2021 18:33:30.681,
clock=e42344a7.e24f1d1b  Thu, Apr 15 2021 19:03:03.884, peer=60391, tc=7,
mintc=3, offset=+1.048514, frequency=+38.158, sys_jitter=4.679301,
clk_jitter=1.975, clk_wander=0.060

Cool_Corona · Apr 16, 2021, 12:32 PM

A couple of the connections we have.

Cool_Corona · Apr 16, 2021, 1:14 PM

Try changing pools

Changed mine from dk, se, no to 0,1,2,3.europe.pool.ntp.org

It bettered the jitter quite considerably.

farmerjohn · Apr 16, 2021, 1:56 PM

@cool_corona
I'll try going back to multiple pool servers (.us.pool.ntp.org) - hopefully that will improve my current situation (status after 19 hours):

[2.5.1-RELEASE][admin@pfsense.home]/var/log: ntpq -pcrv
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.pfsense.pool. .POOL.          16 p    -   64    0    0.000   +0.000   0.000
+tock.sol.net    206.55.64.77     3 u  41m  512   20   37.120   +1.937   1.621
-4.53.160.75 (ns 75.76.123.222    3 u 1722  512   70   26.891   +3.313   1.226
*frigg.fancube.c 107.46.198.112   2 u 1581  512   44   43.347   +3.275   2.363

associd=0 status=0618 leap_none, sync_ntp, 1 event, no_sys_peer,
version="ntpd 4.2.8p15@1.3728-o Fri Feb  5 22:07:56 UTC 2021 (1)",
processor="amd64", system="FreeBSD/12.2-STABLE", leap=00, stratum=3,
precision=-21, rootdelay=49.680, rootdisp=77.172, refid=154.16.245.246,
reftime=e4240ee1.afd0e2c4  Fri, Apr 16 2021  9:25:53.686,
clock=e424150e.f3e02c84  Fri, Apr 16 2021  9:52:14.952, peer=60404, tc=9,
mintc=3, offset=+1.958787, frequency=+37.805, sys_jitter=1.230244,
clk_jitter=3.129, clk_wander=0.225

johnpoz · Apr 16, 2021, 2:24 PM

Make sure your logging your ntp, in the ntp settings.

And look for if ntp is seeing a spike and or resetting for other reasons. Your reaches should be 377, or maybe 1 packet missed now and then, or even a couple..

Those are horrible reach values.. And to be honest, the numbers don't even seem right.. Unless ntp is restarting or you missing a lot of packets..

You shouldn't see numbers like that, unless ntp has reset/restarted and is just starting... Or your loosing a lot of packets in a row..

20 for example, means that you have just restarted and you got 1 packet out of the last 5

10000 = 20 octal

So you have missed the last 4 packets after you got one, and before that you didn't have any.. Or 3 before that was missed 00010000

You really should never see a 20 as it starts up, because it wouldn't count up like that if your getting responses..

Your counts should be
1 = 1
11 = 3
111 = 7
1111 = 17
11111 = 37
111111 = 77
1111111 = 177
11111111 = 377

Your reach values are pointing to a lot of unanswered queries..

your 44 value for example would be this for the last 8 packets..

00100100

So your only getting 1 out of every 3 queries?

70 = 00111000

So you got 3 in a row, and then last 3 nothing..

farmerjohn · Apr 16, 2021, 4:23 PM

@johnpoz
I enabled the two logging options. here are the last 15 - not sure if 'unreachable' lines are a concern or normal.

[2.5.1-RELEASE][admin@pfsense.home]/var/log: cat ntpd.log | tail -15
Apr 16 11:45:37 pfsense ntpd[76193]: Soliciting pool server 142.147.92.5
Apr 16 11:45:41 pfsense ntpd[76193]: 69.89.207.99 101a 8a sys_peer
Apr 16 11:45:41 pfsense ntpd[76193]: 45.79.51.42 0014 84 reachable
Apr 16 11:46:44 pfsense ntpd[76193]: Soliciting pool server 209.50.63.74
Apr 16 11:47:52 pfsense ntpd[76193]: Soliciting pool server 142.147.92.5
Apr 16 11:49:02 pfsense ntpd[76193]: Soliciting pool server 75.76.123.222
Apr 16 11:49:02 pfsense ntpd[76193]: 75.76.123.222 0011 81 mobilize assoc 33714
Apr 16 11:49:03 pfsense ntpd[76193]: Soliciting pool server 184.105.182.7
Apr 16 11:49:03 pfsense ntpd[76193]: 75.76.123.222 0014 84 reachable
Apr 16 11:50:12 pfsense ntpd[76193]: Soliciting pool server 172.86.181.78
Apr 16 11:51:18 pfsense ntpd[76193]: Soliciting pool server 72.14.181.128
Apr 16 11:52:25 pfsense ntpd[76193]: Soliciting pool server 213.32.40.221
Apr 16 11:52:38 pfsense ntpd[76193]: 216.177.181.129 0014 84 reachable
Apr 16 11:54:29 pfsense ntpd[76193]: 69.89.207.99 0613 83 unreachable
Apr 16 11:55:52 pfsense ntpd[76193]: 45.79.51.42 0013 83 unreachable

johnpoz · Apr 16, 2021, 4:40 PM

your prob going to want to look at more than the last 15 entries..

But unreachable isn't good no..

You can tell from your reach numbers your having a real hard time getting responses from the ntp your trying to talk too..

Those numbers your showing for reach values are not indicative of good connectivity.. Take your numbers and convert the octal number to a 8 digit binary and you can see response or no response for the last 8 you have sent.. To that specific IP.. When you are seeing the numbers you are seeing.. You have hard time talking to those ntp servers.. Be it they are not getting your query, or they are not just answering.. But those reach numbers are not good for keeping good time..

edit:
Maybe vs using pool, try some specific one that are in your geographic area
https://support.ntp.org/bin/view/Servers/StratumOneTimeServers
https://support.ntp.org/bin/view/Servers/StratumTwoTimeServers

There are plenty of ntp you can specific point to on the net.. You normally want something close as possible to you, to min the delay. But you for sure need something you can reliably talk too..

Add say time.cloudflare.com

They have like 180+ locations via anycast.. You should be able to hopefully talk to something reliable via that..
https://developers.cloudflare.com/time-services/ntp/usage

farmerjohn · Apr 16, 2021, 5:33 PM

@johnpoz
Changed to 5 individual servers closest to me. Here is status after 20 minutes:

[2.5.1-RELEASE][admin@pfsense.home]/var/log: ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 time.cloudflare .INIT.          16 u    -  512    0    0.000   +0.000   0.000
 rolex.netservic .INIT.          16 u    -  512    0    0.000   +0.000   0.000
 rb.steadfastdns 208.100.0.253    2 u  176   64   74   18.244   +0.095   1.642
 time.skylineser .INIT.          16 u    -  512    0    0.000   +0.000   0.000
*ntp.your.org    204.9.51.33      2 u  701   64    0   20.320   +0.466   0.460

johnpoz · Apr 16, 2021, 6:36 PM

Those are not working ;)

Your isp blocking ntp?

The only one your talking to at all with a reach of 74?

so 00111100..

farmerjohn · Apr 16, 2021, 7:58 PM

@johnpoz
how can i determine if isp is blocking ntp? There is some 123 traffic on WAN interface getting blocked - not sure if due to isp.

I tried to upload image to make things easier for you to troubleshoot, but received notification "something went wrong while parsing server response netgate"

farmerjohn · Apr 16, 2021, 8:03 PM

@johnpoz
upload works on diffeerent browser (firefox-yes, brave no-go)

johnpoz · Apr 16, 2021, 8:10 PM

@farmerjohn said in NTP - poor reach after 20 hours:

There is some 123 traffic on WAN interface getting blocked - not sure if due to isp.

Yeah doesn't sound good.. If that was your return answer from your ntp query..

As to tell if its an isp/internet issue - would sniff on your wan, you would see your ntp queries go out - but never get a response in your sniff. As to that being your isp blocking, or just getting lost in the internet, or the place your trying to talk not answering you - that is the question..

Do you have a vps or something you can fire up, and run ntp server on it - and see if you can talk to that?

If you get a glitch trying to post an image, try again in a few minutes. In the past I have had issues posting images, but try again in few minutes and normally works.

Are you noticing any other issues with your internet connection. Are your states resetting? If your states reset, then yeah return of a ntp query would be blocked because there is no state to allow the traffic back in. Yeah udp is not stateful ;) But pf keeps track of it via a state entry..

edit: Are any of those IPs of NTP servers your trying to talk too?

Doesn't look like it because the source port is not 123.. Looks like they are asking you for ntp.. Do you have your IP listed in the ntp pool or something?

bingo600 · Apr 16, 2021, 8:14 PM

@johnpoz

Just looked up the 123.170.x.x's
I doubt they are NTP ansvers

Seems like CN "probes"

/Bingo

johnpoz · Apr 16, 2021, 8:16 PM

Yeah if it was him asking the source port would be different... That seems to be quite a few probes ;) Maybe that is causing him some grief in trying to get ntp working?

johnpoz · Apr 16, 2021, 8:45 PM

@farmerjohn I think @bingo600 is going to reach out to you via PM with a ntp server you can point too.

I could also have you point directly to my IP, my local ntp server is part of the pool.. This way if know your IP can sniff to see if the queries are even getting to me, etc.

farmerjohn · Apr 16, 2021, 9:35 PM

@johnpoz

Do you have a vps or something you can fire up, and run ntp server on it - and see if you can talk to that?

unfortunately no - only on my LAN, but that probably doesn't count.

Do you have your IP listed in the ntp pool or something?

no, just china and some other unknowns hitting me via 123

Are you noticing any other issues with your internet connection.

One thing I've done recently is enable fq_codel and under general settings / advanced / Firewall & NAT, lowered "TCP Established" to 86400 and "Firewall Optimization" from conservative to normal.

Also setup openvpn on OPT1 and getting a lot of blocked 853 traffic in log, but have not noticed any connectivity issues browsing on openvpn clients.

AKEGEC · Apr 16, 2021, 9:42 PM

@farmerjohn smart move to use Quad9 DNS, it blocks a lot malicious sites.

johnpoz · Apr 16, 2021, 9:45 PM

Those are all out of state blocks.. see the R and Fs those are RST and FIN packets.. Those are what close a session. If the session has already been closed then those would be blocked. etc..

But why are you seeing them on opt1? And your hiding what I assume is your public IP that ends in .37 there? Why would that traffic be inbound to opt1? Is that .37 IP your hiding a rfc1918 address on your internal network?

farmerjohn · Apr 16, 2021, 10:06 PM

@johnpoz

ip that ends in .37 is my public IP.

Why would that traffic be inbound to opt1?

i wish i could answer that.

here is a short sample of tcpdump, I'm terrible at interpreting this, but posted here in case something obvious.

17:45:02.408011 IP (tos 0x0, ttl 255, id 43242, offset 0, flags [none], proto UDP (17), length 76)
    my.public.ip.037.res.spectrum.com.62388 > time-a-b.nist.gov.ntp: [udp sum ok] NTPv4, length 48
	Client, Leap indicator:  (0), Stratum 0 (unspecified), poll 0 (1s), precision 0
	Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec)
	  Reference Timestamp:  0.000000000
	  Originator Timestamp: 0.000000000
	  Receive Timestamp:    0.000000000
	  Transmit Timestamp:   0.000000000
	    Originator - Receive Timestamp:  0.000000000
	    Originator - Transmit Timestamp: 0.000000000
17:45:02.449490 IP (tos 0x0, ttl 48, id 27186, offset 0, flags [none], proto UDP (17), length 76)
    time-a-b.nist.gov.ntp > my.public.ip.037.res.spectrum.com.62388: [udp sum ok] NTPv3, length 48
	Server, Leap indicator:  (0), Stratum 1 (primary reference), poll 13 (8192s), precision -29
	Root Delay: 0.000244, Root dispersion: 0.000488, Reference-ID: NIST
	  Reference Timestamp:  3827598208.000000000 (2021/04/16 17:43:28)
	  Originator Timestamp: 0.000000000
	  Receive Timestamp:    3827598302.430281727 (2021/04/16 17:45:02)
	  Transmit Timestamp:   3827598302.430282955 (2021/04/16 17:45:02)
	    Originator - Receive Timestamp:  3827598302.430281727 (2021/04/16 17:45:02)
	    Originator - Transmit Timestamp: 3827598302.430282955 (2021/04/16 17:45:02)

johnpoz · Apr 16, 2021, 10:24 PM

@farmerjohn said in NTP - poor reach after 20 hours:

i wish i could answer that.

OPT1 is what interface? And why would the source of the traffic "into" opt1 be your public IP? That makes ZERO sense..

How would you be seeing inbound traffic into a lan side interface with your public IP On it???

I don't even see how that would be possible???